[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:16 dist-test-slave-tfpf krb5kdc[29446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:16 dist-test-slave-tfpf krb5kdc[29446](info): set up 2 sockets
Aug 06 02:40:16 dist-test-slave-tfpf krb5kdc[29446](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:40:17.440479 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:17 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:17.560738 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33713
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:17.701510 29463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:17.701817 29463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:17.701933 29463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:17.705917 29463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:17.706003 29463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:17.706082 29463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:17.706140 29463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:17.706187 29463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:17.711445 29463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:33713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:17.712831 29463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:17.714016 29463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:17.726337 29469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:17.726478 29471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:17.726341 29468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:17.727851 29463 server_base.cc:1034] running on GCE node
I20260806 02:40:17.728521 29463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:17.730221 29463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:17.731417 29463 hybrid_clock.cc:648] HybridClock initialized: now 1785984017731389 us; error 64 us; skew 500 ppm
Aug 06 02:40:17 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:17.734422 29463 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:17.735539 29463 webserver.cc:460] Webserver started at http://127.28.191.62:42731/ using document root <none> and password file <none>
I20260806 02:40:17.736433 29463 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:17.736543 29463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:17.736981 29463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:17.739488 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "5e48ec93d87e4041b0f29982c1e9d20c"
format_stamp: "Formatted at 2026-08-06 02:40:17 on dist-test-slave-tfpf"
server_key: "dd7a472d3284d4e548df6d806501055d"
server_key_iv: "9acac40033fde64df7d495bdb3f0340f"
server_key_version: "encryptionkey@0"
I20260806 02:40:17.740101 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "5e48ec93d87e4041b0f29982c1e9d20c"
format_stamp: "Formatted at 2026-08-06 02:40:17 on dist-test-slave-tfpf"
server_key: "dd7a472d3284d4e548df6d806501055d"
server_key_iv: "9acac40033fde64df7d495bdb3f0340f"
server_key_version: "encryptionkey@0"
I20260806 02:40:17.745049 29463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:40:17.748519 29479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:17.750128 29463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:40:17.750342 29463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "5e48ec93d87e4041b0f29982c1e9d20c"
format_stamp: "Formatted at 2026-08-06 02:40:17 on dist-test-slave-tfpf"
server_key: "dd7a472d3284d4e548df6d806501055d"
server_key_iv: "9acac40033fde64df7d495bdb3f0340f"
server_key_version: "encryptionkey@0"
I20260806 02:40:17.750471 29463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:17.766637 29463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:17.770092 29463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:17.770376 29463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:17.784243 29463 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:33713
I20260806 02:40:17.784292 29548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:33713 every 8 connection(s)
I20260806 02:40:17.785573 29463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:17.788889 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29463
I20260806 02:40:17.789135 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:17.790395 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:17.790685 29436 external_mini_cluster.cc:1428] Setting key f7506d0718aefecf62f547aa4f2b2f77
I20260806 02:40:17.799585 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: Bootstrap starting.
I20260806 02:40:17.803228 29549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:17.804322 29549 log.cc:824] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:17.806811 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: No bootstrap required, opened a new log
I20260806 02:40:17.811162 29549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } }
I20260806 02:40:17.811440 29549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:17.811528 29549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e48ec93d87e4041b0f29982c1e9d20c, State: Initialized, Role: FOLLOWER
I20260806 02:40:17.812281 29549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } }
I20260806 02:40:17.812502 29549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:17.812618 29549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:17.812780 29549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:40:17 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:17.814083 29549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } }
I20260806 02:40:17.814759 29549 leader_election.cc:302] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e48ec93d87e4041b0f29982c1e9d20c; no voters:
I20260806 02:40:17.815184 29549 leader_election.cc:288] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:17.815335 29556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:17.815590 29556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [term 1 LEADER]: Becoming Leader. State: Replica: 5e48ec93d87e4041b0f29982c1e9d20c, State: Running, Role: LEADER
I20260806 02:40:17.816068 29556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } }
I20260806 02:40:17.816901 29549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:17.819152 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } } }
I20260806 02:40:17.819283 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:17.821278 29560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e48ec93d87e4041b0f29982c1e9d20c. Latest consensus state: current_term: 1 leader_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e48ec93d87e4041b0f29982c1e9d20c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33713 } } }
I20260806 02:40:17.821379 29560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:17.821877 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:17.795405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.28.191.62:33713)
0806 02:40:17.795915 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:17.795933 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:17.799908 (+ 3975us) server_negotiation.cc:407] Connection header received
0806 02:40:17.803381 (+ 3473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:17.803404 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:17.803792 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:17.804133 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:17.806675 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:17.807796 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:17.809039 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:17.809585 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:17.809784 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:17.814557 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:17.814602 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:17.814615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:17.814644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:17.817103 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:17.817897 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:17.817906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:17.817915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:17.818093 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:17.818609 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:17.818613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:17.818615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:17.819408 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:17.819572 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:17.820530 (+ 958us) server_negotiation.cc:299] Negotiation successful
0806 02:40:17.820755 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":165,"threads_started":1}
W20260806 02:40:17.825171 29577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:40:17.825304 29577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:40:17.825387 29578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:17.826582 29578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:17.833468 29578 catalog_manager.cc:1347] Generated new cluster ID: 69deadcad4224e578743b37bb4e20602
I20260806 02:40:17.833565 29578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:17.844879 29578 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:17.845871 29578 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:17.851291 29578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e48ec93d87e4041b0f29982c1e9d20c: Generated new TSK 0
I20260806 02:40:17.852048 29578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:17.884382 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:18.036262 29586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:18.036561 29586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:18.036638 29586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:18.040758 29586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:18.040848 29586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:18.040941 29586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:18.046270 29586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:18.047375 29586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:18.048368 29586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:18.059186 29593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:18.059190 29591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:18.059190 29595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:18.059542 29586 server_base.cc:1034] running on GCE node
I20260806 02:40:18.060045 29586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:18.060736 29586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:18.061916 29586 hybrid_clock.cc:648] HybridClock initialized: now 1785984018061876 us; error 63 us; skew 500 ppm
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:18.065358 29586 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:18.066176 29586 webserver.cc:460] Webserver started at http://127.28.191.1:34203/ using document root <none> and password file <none>
I20260806 02:40:18.066738 29586 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:18.066828 29586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:18.067060 29586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:18.069177 29586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "fa56abffdd1845e582364e4154461b67"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "c1b1889d4b2cb23e4fadbb358972fece"
server_key_iv: "f4f3931da8a8deac85db04d1c24ad7e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.069751 29586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "fa56abffdd1845e582364e4154461b67"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "c1b1889d4b2cb23e4fadbb358972fece"
server_key_iv: "f4f3931da8a8deac85db04d1c24ad7e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.073896 29586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:40:18.076297 29610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.077598 29586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:18.077737 29586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "fa56abffdd1845e582364e4154461b67"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "c1b1889d4b2cb23e4fadbb358972fece"
server_key_iv: "f4f3931da8a8deac85db04d1c24ad7e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.077849 29586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:18.091562 29586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:18.094964 29586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:18.095162 29586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:18.095868 29586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:18.096923 29586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:18.096974 29586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.097045 29586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:18.097095 29586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 02:40:18.109517 29586 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:44385
I20260806 02:40:18.109599 29793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:44385 every 8 connection(s)
I20260806 02:40:18.110430 29586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:18.111872 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29586
I20260806 02:40:18.111982 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:18.112282 29436 external_mini_cluster.cc:1428] Setting key eb9ba2b7610698146587911fa358d4e4
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:40:18.126427 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.112872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:57967 (local address 127.28.191.62:33713)
0806 02:40:18.113018 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:18.113023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:18.113953 (+ 930us) server_negotiation.cc:407] Connection header received
0806 02:40:18.115687 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:18.115690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:18.115746 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:18.115847 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:18.117455 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.118120 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.119399 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.119639 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.119747 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.122304 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:18.122308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:18.122310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:18.122338 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:18.123858 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.124449 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.124452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.124453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.124541 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.124938 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.124941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.124942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.125196 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:18.125303 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:18.126017 (+ 714us) server_negotiation.cc:299] Negotiation successful
0806 02:40:18.126171 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:40:18.127022 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.113333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33713 (local address 127.28.191.1:57967)
0806 02:40:18.113787 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:18.113839 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:18.115474 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:18.116025 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:18.116032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:18.116526 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:18.117314 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.117332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.118244 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.118247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.119271 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.119281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.119801 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.119806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.119981 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.120625 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:18.120642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:18.122150 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:18.124060 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.124065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.124077 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.124343 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.124627 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.124629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.124631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.124804 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.125346 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:18.125355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:18.125691 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 02:40:18.125904 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 02:40:18.126178 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260806 02:40:18.128451 29794 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33713
I20260806 02:40:18.128700 29794 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:18.129390 29794 heartbeater.cc:511] Master 127.28.191.62:33713 requested a full tablet report, sending...
I20260806 02:40:18.131600 29498 ts_manager.cc:194] Registered new tserver with Master: fa56abffdd1845e582364e4154461b67 (127.28.191.1:44385)
I20260806 02:40:18.132936 29498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:57967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:18.141870 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:18.246829 29801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:18.247071 29801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:18.247164 29801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:18.251179 29801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:18.251260 29801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:18.251348 29801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:18.256160 29801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:18.257128 29801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:18.258087 29801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:18.267252 29807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:18.267304 29808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:18.267288 29801 server_base.cc:1034] running on GCE node
W20260806 02:40:18.267248 29812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:18.267822 29801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:18.268412 29801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:18.269562 29801 hybrid_clock.cc:648] HybridClock initialized: now 1785984018269532 us; error 41 us; skew 500 ppm
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:18.272341 29801 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:18.273286 29801 webserver.cc:460] Webserver started at http://127.28.191.2:42399/ using document root <none> and password file <none>
I20260806 02:40:18.273923 29801 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:18.274017 29801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:18.274232 29801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:18.276146 29801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "cd1c26caa6cc40a5ba341c5632d7d170"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "08c13d2f26de2ae7fd7fa25a40bcdc5f"
server_key_iv: "53dda0ba1447de11b6d85543ee6bd624"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.276746 29801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "cd1c26caa6cc40a5ba341c5632d7d170"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "08c13d2f26de2ae7fd7fa25a40bcdc5f"
server_key_iv: "53dda0ba1447de11b6d85543ee6bd624"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.280306 29801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:18.282693 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.283857 29801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:18.283994 29801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "cd1c26caa6cc40a5ba341c5632d7d170"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "08c13d2f26de2ae7fd7fa25a40bcdc5f"
server_key_iv: "53dda0ba1447de11b6d85543ee6bd624"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.284111 29801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:18.293570 29801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:18.296641 29801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:18.296864 29801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:18.297479 29801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:18.298254 29801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:18.298324 29801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.298394 29801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:18.298445 29801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.307515 29801 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:41037
I20260806 02:40:18.307540 29986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:41037 every 8 connection(s)
I20260806 02:40:18.308262 29801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:18.317768 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29801
I20260806 02:40:18.317905 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:18.318293 29436 external_mini_cluster.cc:1428] Setting key 22eb17050cf400cdd75588706a96f675
I20260806 02:40:18.322331 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.310007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:41587 (local address 127.28.191.62:33713)
0806 02:40:18.310150 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:18.310154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:18.310810 (+ 656us) server_negotiation.cc:407] Connection header received
0806 02:40:18.312160 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:18.312165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:18.312234 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:18.312336 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:18.313541 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.314086 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.315239 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.315494 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.315624 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.318199 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:18.318203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:18.318206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:18.318233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:18.320053 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.320574 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.320578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.320580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.320669 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.321011 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.321015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.321016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.321225 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:18.321336 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:18.322026 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 02:40:18.322158 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:18.323647 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.310226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33713 (local address 127.28.191.2:41587)
0806 02:40:18.310659 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:18.310699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:18.311920 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:18.312501 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:18.312508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:18.312848 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:18.313370 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.313386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.314231 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.314235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.315092 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.315102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.315662 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.315666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.315850 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.316349 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:18.316365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:18.317929 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:18.320169 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.320174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.320188 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.320439 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.320762 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.320766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.320768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.320902 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.321451 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:18.321455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:18.321741 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 02:40:18.322492 (+ 751us) client_negotiation.cc:240] Negotiation successful
0806 02:40:18.322844 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260806 02:40:18.324807 29987 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33713
I20260806 02:40:18.325105 29987 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:18.325553 29987 heartbeater.cc:511] Master 127.28.191.62:33713 requested a full tablet report, sending...
I20260806 02:40:18.326594 29498 ts_manager.cc:194] Registered new tserver with Master: cd1c26caa6cc40a5ba341c5632d7d170 (127.28.191.2:41037)
I20260806 02:40:18.327138 29498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:41587
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:18.345423 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:18.453794 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:18.454115 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:18.454178 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:18.458214 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:18.458261 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:18.458315 29992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:18.463124 29992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:18.464056 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:18.464978 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:18.474186 29997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:18.474285 29998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:18.474227 30000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:18.474359 29992 server_base.cc:1034] running on GCE node
I20260806 02:40:18.474862 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:18.475447 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:18.476605 29992 hybrid_clock.cc:648] HybridClock initialized: now 1785984018476578 us; error 36 us; skew 500 ppm
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:18.479344 29992 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:18.480145 29992 webserver.cc:460] Webserver started at http://127.28.191.3:41929/ using document root <none> and password file <none>
I20260806 02:40:18.480718 29992 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:18.480778 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:18.481014 29992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:18.483014 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "a8bae78a77694c38ac138da6b802247a"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "6716671f8e089b2e198aebcc16dd88e3"
server_key_iv: "df2a5c4573462ad772473702836b897f"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.483536 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "a8bae78a77694c38ac138da6b802247a"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "6716671f8e089b2e198aebcc16dd88e3"
server_key_iv: "df2a5c4573462ad772473702836b897f"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.486985 29992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 02:40:18.489218 30009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.490381 29992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:18.490518 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "a8bae78a77694c38ac138da6b802247a"
format_stamp: "Formatted at 2026-08-06 02:40:18 on dist-test-slave-tfpf"
server_key: "6716671f8e089b2e198aebcc16dd88e3"
server_key_iv: "df2a5c4573462ad772473702836b897f"
server_key_version: "encryptionkey@0"
I20260806 02:40:18.490625 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:18.513128 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:18.515620 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:18.515808 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:18.516355 29992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:18.517212 29992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:18.517261 29992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.517330 29992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:18.517370 29992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:18.526316 29992 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:40117
I20260806 02:40:18.526342 30167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:40117 every 8 connection(s)
I20260806 02:40:18.527160 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:18.531966 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29992
I20260806 02:40:18.532100 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:18.532354 29436 external_mini_cluster.cc:1428] Setting key 4d3c4d35a422b10433a0c1e63cf7a2c9
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:18.540863 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.528952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:51901 (local address 127.28.191.62:33713)
0806 02:40:18.529098 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:18.529101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:18.529957 (+ 856us) server_negotiation.cc:407] Connection header received
0806 02:40:18.531299 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:18.531303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:18.531364 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:18.531472 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:18.532766 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.533563 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.535156 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.535404 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.535535 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.537987 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:18.537991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:18.537994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:18.538017 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:18.539196 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.539626 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.539629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.539630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.539698 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.539983 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.539985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.539986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.540135 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:18.540204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:18.540656 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 02:40:18.540746 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:40:18.541675 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.529379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33713 (local address 127.28.191.3:51901)
0806 02:40:18.529807 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:18.529848 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:18.531088 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:18.531670 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:18.531677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:18.532052 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:18.532605 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.532620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.533692 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.533695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.535009 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:18.535020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.535560 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.535563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:18.535694 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.536281 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:18.536297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:18.537827 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:18.539297 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.539302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.539312 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.539543 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.539775 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:18.539777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:18.539778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:18.539898 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:18.540215 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:18.540219 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:18.540415 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 02:40:18.540609 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 02:40:18.540838 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260806 02:40:18.542766 30168 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33713
I20260806 02:40:18.542999 30168 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:18.543480 30168 heartbeater.cc:511] Master 127.28.191.62:33713 requested a full tablet report, sending...
I20260806 02:40:18.544423 29498 ts_manager.cc:194] Registered new tserver with Master: a8bae78a77694c38ac138da6b802247a (127.28.191.3:40117)
I20260806 02:40:18.544906 29498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:51901
I20260806 02:40:18.547211 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:18.574229 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.564693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.28.191.1:44385)
0806 02:40:18.565052 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:18.565104 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:18.565136 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:40:18.565184 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:18.565189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:18.565351 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:18.565465 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:18.566305 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.567025 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.567836 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.568178 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.568279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.570782 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:18.570792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:18.570804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:18.570827 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:18.572487 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.572853 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.572858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.572859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.572972 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.573302 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.573305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.573306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.573539 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:18.573685 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:18.573837 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:40:18.573987 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
W20260806 02:40:18.575326 29649 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52196
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:18.602219 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:18.594071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.28.191.1:44385)
0806 02:40:18.594227 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:18.594231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:18.594449 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:40:18.594646 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:18.594650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:18.594715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:18.594815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:18.595608 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.596115 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.596911 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:18.597220 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:18.597298 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:18.599583 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:18.599586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:18.599588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:18.599610 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:18.600817 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.601249 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.601251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.601253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.601326 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:18.601534 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:18.601536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:18.601537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:18.601677 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:18.601768 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:18.601949 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:40:18.602042 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:40:18.603286 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29586
I20260806 02:40:18.612905 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29801
I20260806 02:40:18.620952 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29992
I20260806 02:40:18.628700 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29463
2026-08-06T02:40:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2388 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[30187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[30187](info): set up 2 sockets
Aug 06 02:40:18 dist-test-slave-tfpf krb5kdc[30187](info): commencing operation
krb5kdc: starting...
W20260806 02:40:19.818717 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:19 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:19.937232 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35789
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:20.045185 30205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:20.045439 30205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:20.045506 30205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:20.049456 30205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:20.049521 30205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:20.049556 30205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:20.049605 30205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:20.049644 30205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:20.054426 30205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:20.055326 30205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:20.056246 30205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:20.064388 30214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.064388 30213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.064378 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:20.064720 30205 server_base.cc:1034] running on GCE node
I20260806 02:40:20.065308 30205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:20.066324 30205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:20.067593 30205 hybrid_clock.cc:648] HybridClock initialized: now 1785984020067556 us; error 39 us; skew 500 ppm
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:20.070367 30205 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:20.071259 30205 webserver.cc:460] Webserver started at http://127.28.191.62:43941/ using document root <none> and password file <none>
I20260806 02:40:20.071835 30205 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:20.071930 30205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:20.072175 30205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:20.074157 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "b1ec06aa13504c289f182e895e6922fc"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "1c9f064b6b3835ec40895d1c7e075f88"
server_key_iv: "2c91192b6a882fc43982a424cec3d79c"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.074736 30205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "b1ec06aa13504c289f182e895e6922fc"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "1c9f064b6b3835ec40895d1c7e075f88"
server_key_iv: "2c91192b6a882fc43982a424cec3d79c"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.078581 30205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:40:20.080935 30230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.082105 30205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:20.082243 30205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "b1ec06aa13504c289f182e895e6922fc"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "1c9f064b6b3835ec40895d1c7e075f88"
server_key_iv: "2c91192b6a882fc43982a424cec3d79c"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.082351 30205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:20.098948 30205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:20.101646 30205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:20.101847 30205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:20.109076 30205 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35789
I20260806 02:40:20.109138 30308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35789 every 8 connection(s)
I20260806 02:40:20.110001 30205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:20.112530 30309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:20.113265 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 30205
I20260806 02:40:20.113387 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:20.113641 29436 external_mini_cluster.cc:1428] Setting key 36b52c6141121fc66aa37736542d75a2
I20260806 02:40:20.118316 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: Bootstrap starting.
I20260806 02:40:20.120338 30309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:20.121169 30309 log.cc:824] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:20.122920 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: No bootstrap required, opened a new log
I20260806 02:40:20.126595 30309 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } }
I20260806 02:40:20.126822 30309 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:20.126895 30309 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ec06aa13504c289f182e895e6922fc, State: Initialized, Role: FOLLOWER
I20260806 02:40:20.127347 30309 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } }
I20260806 02:40:20.127441 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.114954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.28.191.62:35789)
0806 02:40:20.115390 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.115411 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.115445 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:40:20.116098 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.116115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.116409 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.116739 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.117635 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.118641 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.119472 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.119939 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.120120 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.122754 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.122760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.122772 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.122799 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.124474 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.124936 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.124942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.124946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.125112 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.125423 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.125427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.125428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.125804 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.126070 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.126326 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.126619 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
I20260806 02:40:20.127483 30309 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:20.127574 30309 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:20.127671 30309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:20.128476 30309 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } }
I20260806 02:40:20.128818 30309 leader_election.cc:302] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ec06aa13504c289f182e895e6922fc; no voters:
I20260806 02:40:20.129149 30309 leader_election.cc:288] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:20.129243 30318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:20.129418 30318 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [term 1 LEADER]: Becoming Leader. State: Replica: b1ec06aa13504c289f182e895e6922fc, State: Running, Role: LEADER
I20260806 02:40:20.129696 30318 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } }
I20260806 02:40:20.130349 30309 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:20.130873 30318 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ec06aa13504c289f182e895e6922fc. Latest consensus state: current_term: 1 leader_uuid: "b1ec06aa13504c289f182e895e6922fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } } }
I20260806 02:40:20.130988 30318 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:20.131616 30320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1ec06aa13504c289f182e895e6922fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ec06aa13504c289f182e895e6922fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35789 } } }
I20260806 02:40:20.131735 30320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc [sys.catalog]: This master's current role is: LEADER
W20260806 02:40:20.135021 30340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:40:20.135097 30340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:40:20.135177 30325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:20.135967 30325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:20.142413 30325 catalog_manager.cc:1347] Generated new cluster ID: fed0613d564e47a7a05d9f8642f311e4
I20260806 02:40:20.142482 30325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:20.150492 30325 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:20.151620 30325 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:20.158829 30325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1ec06aa13504c289f182e895e6922fc: Generated new TSK 0
I20260806 02:40:20.159368 30325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:20.195466 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:20.322222 30348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:20.322455 30348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:20.322518 30348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:20.326154 30348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:20.326198 30348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:20.326247 30348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:20.331138 30348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:20.332026 30348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:20.332892 30348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:20.342110 30359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:20.342281 30348 server_base.cc:1034] running on GCE node
W20260806 02:40:20.342123 30361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.342141 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:20.342828 30348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:20.343443 30348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:20.344626 30348 hybrid_clock.cc:648] HybridClock initialized: now 1785984020344610 us; error 26 us; skew 500 ppm
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:20.347311 30348 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:20.348090 30348 webserver.cc:460] Webserver started at http://127.28.191.1:41875/ using document root <none> and password file <none>
I20260806 02:40:20.348608 30348 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:20.348686 30348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:20.348889 30348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:20.350992 30348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "4ec881b239b7446488480924acc29572"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "baee3ccc5d311d61a10f1d25c9d34e36"
server_key_iv: "8aab8c8d8e96edc59ba0af995898dcd1"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.351563 30348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "4ec881b239b7446488480924acc29572"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "baee3ccc5d311d61a10f1d25c9d34e36"
server_key_iv: "8aab8c8d8e96edc59ba0af995898dcd1"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.355118 30348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:20.357435 30375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.358554 30348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:20.358675 30348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "4ec881b239b7446488480924acc29572"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "baee3ccc5d311d61a10f1d25c9d34e36"
server_key_iv: "8aab8c8d8e96edc59ba0af995898dcd1"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.358773 30348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:20.372623 30348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:20.375167 30348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:20.375355 30348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:20.375844 30348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:20.376601 30348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:20.376662 30348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.376729 30348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:20.376791 30348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.385172 30348 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:33013
I20260806 02:40:20.385181 30532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:33013 every 8 connection(s)
I20260806 02:40:20.385985 30348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:20.392166 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 30348
I20260806 02:40:20.392277 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:20.392537 29436 external_mini_cluster.cc:1428] Setting key 90c416e6771b374b8b25370fe3f9641c
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:20.398788 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.387503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:39577 (local address 127.28.191.62:35789)
0806 02:40:20.387623 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.387627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.388406 (+ 779us) server_negotiation.cc:407] Connection header received
0806 02:40:20.389595 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.389597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.389643 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.389724 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.390767 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.391291 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.392339 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.392661 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.392763 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.395081 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.395087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.395092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.395121 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.396531 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.397164 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.397171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.397173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.397249 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.397660 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.397663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.397664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.397828 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.397915 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.398517 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.398646 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:40:20.399897 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.387847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35789 (local address 127.28.191.1:39577)
0806 02:40:20.388259 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.388295 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.389420 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.389883 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.389889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.390195 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:20.390604 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.390635 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.391392 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.391395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.392211 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.392219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.392790 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.392792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.392923 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.393482 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:20.393499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:20.394915 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:20.396655 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.396662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.396675 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.397037 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.397343 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.397347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.397351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.397559 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.397941 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:20.397949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:20.398267 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.398468 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.398775 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":133,"threads_started":1}
I20260806 02:40:20.401121 30533 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35789
I20260806 02:40:20.401355 30533 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:20.401803 30533 heartbeater.cc:511] Master 127.28.191.62:35789 requested a full tablet report, sending...
I20260806 02:40:20.403350 30258 ts_manager.cc:194] Registered new tserver with Master: 4ec881b239b7446488480924acc29572 (127.28.191.1:33013)
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:20.404526 30258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:39577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:20.419703 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:20.527101 30543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:20.527401 30543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:20.527479 30543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:20.531631 30543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:20.531708 30543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:20.531818 30543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:20.537002 30543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:20.538108 30543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:20.539119 30543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:20.550720 30553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.551440 30550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.551697 30556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:20.552274 30543 server_base.cc:1034] running on GCE node
I20260806 02:40:20.552702 30543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:20.553437 30543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:20.554608 30543 hybrid_clock.cc:648] HybridClock initialized: now 1785984020554592 us; error 29 us; skew 500 ppm
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:20.557276 30543 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:20.558122 30543 webserver.cc:460] Webserver started at http://127.28.191.2:36611/ using document root <none> and password file <none>
I20260806 02:40:20.558670 30543 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:20.558727 30543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:20.558934 30543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:20.560827 30543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "3a446f4525d30ce30617a069df48efa3"
server_key_iv: "73aa6ce9f4acd0e202673819dba37207"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.561439 30543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "3a446f4525d30ce30617a069df48efa3"
server_key_iv: "73aa6ce9f4acd0e202673819dba37207"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.565387 30543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:40:20.567816 30563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.569238 30543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:20.569376 30543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "3a446f4525d30ce30617a069df48efa3"
server_key_iv: "73aa6ce9f4acd0e202673819dba37207"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.569487 30543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:20.591368 30543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:20.593986 30543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:20.594144 30543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:20.594636 30543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:20.595407 30543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:20.595453 30543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.595490 30543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:20.595511 30543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.605216 30543 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:46579
I20260806 02:40:20.605240 30732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:46579 every 8 connection(s)
I20260806 02:40:20.606043 30543 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:20.615232 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 30543
I20260806 02:40:20.615352 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:20.615655 29436 external_mini_cluster.cc:1428] Setting key 106e456f0ff926c92c3d8a43f562c589
I20260806 02:40:20.619251 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.607757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:60935 (local address 127.28.191.62:35789)
0806 02:40:20.607892 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.607896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.608670 (+ 774us) server_negotiation.cc:407] Connection header received
0806 02:40:20.609919 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.609923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.609969 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.610040 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.611163 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.611669 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.612719 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.612931 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.613029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.615545 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.615549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.615552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.615580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.617276 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.617748 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.617753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.617755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.617844 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.618179 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.618184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.618186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.618378 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.618480 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.618966 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.619084 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:40:20.620008 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.608115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35789 (local address 127.28.191.2:60935)
0806 02:40:20.608525 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.608567 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.609721 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.610210 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.610216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.610580 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:20.611032 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.611054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.611770 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.611772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.612610 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.612619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.613109 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.613118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.613262 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.613917 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:20.613933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:20.615366 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:20.617380 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.617391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.617401 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.617645 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.617947 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.617949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.617951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.618084 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.618478 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:20.618485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:20.618708 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.618919 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.619224 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260806 02:40:20.621109 30735 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35789
I20260806 02:40:20.621349 30735 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:20.621937 30735 heartbeater.cc:511] Master 127.28.191.62:35789 requested a full tablet report, sending...
I20260806 02:40:20.623061 30258 ts_manager.cc:194] Registered new tserver with Master: fc7f126b881a4b9aa1fbb5d10c41fde9 (127.28.191.2:46579)
I20260806 02:40:20.623916 30258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:60935
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:20.644164 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:20.756524 30741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:20.756793 30741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:20.756839 30741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:20.760820 30741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:20.760871 30741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:20.760931 30741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:20.766021 30741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:20.766956 30741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:20.767863 30741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:20.777386 30749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.777411 30747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:20.777401 30746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:20.777666 30741 server_base.cc:1034] running on GCE node
I20260806 02:40:20.778170 30741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:20.778772 30741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:20.779950 30741 hybrid_clock.cc:648] HybridClock initialized: now 1785984020779937 us; error 35 us; skew 500 ppm
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:20.782765 30741 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:20.783571 30741 webserver.cc:460] Webserver started at http://127.28.191.3:37039/ using document root <none> and password file <none>
I20260806 02:40:20.784108 30741 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:20.784194 30741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:20.784411 30741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:20.786419 30741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "40f53301cb6f4b8caa91705d704132ad"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "43432b0eb3bef9503cc60c9774e41705"
server_key_iv: "0cf3058c16581505f105747fdf6cc5e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.787014 30741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "40f53301cb6f4b8caa91705d704132ad"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "43432b0eb3bef9503cc60c9774e41705"
server_key_iv: "0cf3058c16581505f105747fdf6cc5e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.790508 30741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:40:20.792723 30757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.793895 30741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:20.794024 30741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "40f53301cb6f4b8caa91705d704132ad"
format_stamp: "Formatted at 2026-08-06 02:40:20 on dist-test-slave-tfpf"
server_key: "43432b0eb3bef9503cc60c9774e41705"
server_key_iv: "0cf3058c16581505f105747fdf6cc5e7"
server_key_version: "encryptionkey@0"
I20260806 02:40:20.794137 30741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:20.811190 30741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:20.814461 30741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:20.814661 30741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:20.815255 30741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:20.816097 30741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:20.816144 30741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.816207 30741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:20.816242 30741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:20.824741 30741 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:43109
I20260806 02:40:20.824765 30922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:43109 every 8 connection(s)
I20260806 02:40:20.825605 30741 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:20.830246 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 30741
I20260806 02:40:20.830366 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:20.830642 29436 external_mini_cluster.cc:1428] Setting key 696901249994d37a16ec26bd5ece3d2f
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984020, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:20.838717 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.827271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:49741 (local address 127.28.191.62:35789)
0806 02:40:20.827408 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.827412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.828175 (+ 763us) server_negotiation.cc:407] Connection header received
0806 02:40:20.829475 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.829479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.829536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.829634 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.830963 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.831693 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.833143 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.833335 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.833408 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.835598 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.835600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.835602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.835626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.837020 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.837502 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.837504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.837506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.837584 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.837860 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.837862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.837863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.838035 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.838119 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.838513 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.838605 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:40:20.839443 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.827627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35789 (local address 127.28.191.3:49741)
0806 02:40:20.828041 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.828082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.829275 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.829805 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.829812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.830202 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:20.830822 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.830839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.831862 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.831865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.832987 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.832996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.833459 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.833462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.833609 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.834092 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:20.834107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:20.835455 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:20.837169 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.837173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.837183 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.837416 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.837657 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:20.837659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:20.837661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:20.837786 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:20.838126 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:20.838131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:20.838329 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.838471 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.838691 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260806 02:40:20.840413 30923 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35789
I20260806 02:40:20.840631 30923 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:20.841156 30923 heartbeater.cc:511] Master 127.28.191.62:35789 requested a full tablet report, sending...
I20260806 02:40:20.842116 30258 ts_manager.cc:194] Registered new tserver with Master: 40f53301cb6f4b8caa91705d704132ad (127.28.191.3:43109)
I20260806 02:40:20.842633 30258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:49741
I20260806 02:40:20.844352 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:40:20.854846 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.848189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.28.191.62:35789)
0806 02:40:20.848332 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.848336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.848421 (+ 85us) server_negotiation.cc:407] Connection header received
0806 02:40:20.848513 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.848515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.848550 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.848612 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.849287 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.849739 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.850452 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.850646 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.850737 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.851976 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.851978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.851980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.851996 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.853228 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.853629 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.853631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.853632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.853697 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.853906 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.853908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.853909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.854104 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.854202 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.854466 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.854671 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:40:20.859740 30258 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:20.862891 30258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:20.877884 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.871745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:33013 (local address 127.0.0.1:58704)
0806 02:40:20.872194 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.872245 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.872374 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.873185 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.873192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.873211 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:20.873365 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.873376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.874921 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.874924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.876349 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.876361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877479 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.877585 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877606 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.877687 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.877739 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":68,"threads_started":1}
I20260806 02:40:20.878116 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.872840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.28.191.3:43109)
0806 02:40:20.873611 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.873624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.873645 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:40:20.873709 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.873714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.873873 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.874014 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:20.874188 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.875171 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.876454 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877036 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877188 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877395 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.877851 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.877969 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":123,"threads_started":1}
I20260806 02:40:20.878144 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.871872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.28.191.1:33013)
0806 02:40:20.872528 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.872557 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.872579 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:40:20.872657 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.872663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.872856 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.873011 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:20.873609 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.874778 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.876498 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877369 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877474 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877608 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.877852 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.877987 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":128,"threads_started":1}
I20260806 02:40:20.878250 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.871844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:46579 (local address 127.0.0.1:55242)
0806 02:40:20.872230 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.872245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.873157 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.873772 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.873775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.873792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:20.873890 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.873899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.875225 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.875228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.876372 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.876379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877703 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.877932 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877967 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.878050 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.878110 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":26,"thread_start_us":97,"threads_started":1}
I20260806 02:40:20.878750 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.872845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.28.191.2:46579)
0806 02:40:20.873323 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.873338 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.873361 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:40:20.873434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.873441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.873644 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.873780 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:20.874253 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.875082 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.876498 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877369 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877474 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877608 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.878510 (+ 902us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.878607 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":165,"threads_started":1}
I20260806 02:40:20.877765 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.872444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:43109 (local address 127.0.0.1:44436)
0806 02:40:20.873296 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.873310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.873399 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.874014 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.874017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.874033 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:20.874104 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.874108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.875326 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.875329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.876332 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.876343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.877186 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.877190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.877314 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.877354 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.877472 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.877588 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":38,"thread_start_us":98,"threads_started":1}
I20260806 02:40:20.881625 30636 tablet_service.cc:1467] Processing CreateTablet for tablet d06a181fea094f0ba649fbe9b8fbed91 (DEFAULT_TABLE table=test-table [id=553bff869e0b47c380c43852695beb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:20.881664 30439 tablet_service.cc:1467] Processing CreateTablet for tablet d06a181fea094f0ba649fbe9b8fbed91 (DEFAULT_TABLE table=test-table [id=553bff869e0b47c380c43852695beb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:20.882288 30828 tablet_service.cc:1467] Processing CreateTablet for tablet d06a181fea094f0ba649fbe9b8fbed91 (DEFAULT_TABLE table=test-table [id=553bff869e0b47c380c43852695beb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:20.882723 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06a181fea094f0ba649fbe9b8fbed91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:20.882766 30439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06a181fea094f0ba649fbe9b8fbed91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:20.883219 30828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06a181fea094f0ba649fbe9b8fbed91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:20.889151 30943 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: Bootstrap starting.
I20260806 02:40:20.890369 30949 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: Bootstrap starting.
I20260806 02:40:20.891697 30943 tablet_bootstrap.cc:654] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:20.892056 30949 tablet_bootstrap.cc:654] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:20.892450 30943 log.cc:824] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:20.893117 30946 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: Bootstrap starting.
I20260806 02:40:20.893293 30949 log.cc:824] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:20.895026 30946 tablet_bootstrap.cc:654] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:20.895054 30949 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: No bootstrap required, opened a new log
I20260806 02:40:20.895231 30949 ts_tablet_manager.cc:1397] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:40:20.896006 30946 log.cc:824] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:20.897541 30943 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: No bootstrap required, opened a new log
I20260806 02:40:20.897733 30943 ts_tablet_manager.cc:1397] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 02:40:20.897915 30946 tablet_bootstrap.cc:492] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: No bootstrap required, opened a new log
I20260806 02:40:20.898108 30946 ts_tablet_manager.cc:1397] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:40:20.898778 30949 raft_consensus.cc:348] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.899065 30949 raft_consensus.cc:374] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:20.899124 30949 raft_consensus.cc:729] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f53301cb6f4b8caa91705d704132ad, State: Initialized, Role: FOLLOWER
I20260806 02:40:20.899564 30949 consensus_queue.cc:260] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.900300 30923 heartbeater.cc:503] Master 127.28.191.62:35789 was elected leader, sending a full tablet report...
I20260806 02:40:20.901216 30949 ts_tablet_manager.cc:1428] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:40:20.901764 30946 raft_consensus.cc:348] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.902001 30946 raft_consensus.cc:374] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:20.902063 30946 raft_consensus.cc:729] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec881b239b7446488480924acc29572, State: Initialized, Role: FOLLOWER
I20260806 02:40:20.902426 30946 consensus_queue.cc:260] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.902490 30943 raft_consensus.cc:348] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.902782 30943 raft_consensus.cc:374] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:20.902849 30943 raft_consensus.cc:729] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7f126b881a4b9aa1fbb5d10c41fde9, State: Initialized, Role: FOLLOWER
I20260806 02:40:20.903095 30533 heartbeater.cc:503] Master 127.28.191.62:35789 was elected leader, sending a full tablet report...
I20260806 02:40:20.903304 30946 ts_tablet_manager.cc:1428] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 02:40:20.903295 30943 consensus_queue.cc:260] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.904021 30735 heartbeater.cc:503] Master 127.28.191.62:35789 was elected leader, sending a full tablet report...
I20260806 02:40:20.904732 30943 ts_tablet_manager.cc:1428] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:40:20.965896 30954 raft_consensus.cc:482] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:20.966194 30954 raft_consensus.cc:504] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.967720 30954 leader_election.cc:288] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec881b239b7446488480924acc29572 (127.28.191.1:33013), fc7f126b881a4b9aa1fbb5d10c41fde9 (127.28.191.2:46579)
I20260806 02:40:20.972734 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.968112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:39315 (local address 127.28.191.1:33013)
0806 02:40:20.968255 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.968260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.968306 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:40:20.968584 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.968587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.968645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.968719 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:20.969196 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.969945 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.971460 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.972071 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.972194 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.972257 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.972560 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.972625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:40:20.972734 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.967949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:33013 (local address 127.28.191.3:39315)
0806 02:40:20.968217 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.968259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.968387 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.968902 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.968905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.968923 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:20.969043 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.969049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.970081 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.970084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.971306 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.971318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.972324 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.972328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.972431 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.972450 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.972543 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.972610 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 02:40:20.973444 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.968563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:46579 (local address 127.28.191.3:50421)
0806 02:40:20.968952 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:20.968970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:20.969120 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:20.969565 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:20.969587 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:20.969605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:20.969725 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.969731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.970800 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.970804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.972105 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:20.972120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.973032 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.973035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:20.973159 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.973198 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:40:20.973275 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:40:20.973322 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":109,"threads_started":1}
I20260806 02:40:20.973538 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.968269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:50421 (local address 127.28.191.2:46579)
0806 02:40:20.968425 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.968432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.969088 (+ 656us) server_negotiation.cc:407] Connection header received
0806 02:40:20.969317 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.969322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.969379 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.969454 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:20.969834 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.970616 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.972264 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.972765 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.972876 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.972930 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.973302 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.973413 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:40:20.973526 30470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06a181fea094f0ba649fbe9b8fbed91" candidate_uuid: "40f53301cb6f4b8caa91705d704132ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec881b239b7446488480924acc29572" is_pre_election: true
I20260806 02:40:20.973946 30470 raft_consensus.cc:2454] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f53301cb6f4b8caa91705d704132ad in term 0.
I20260806 02:40:20.974457 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06a181fea094f0ba649fbe9b8fbed91" candidate_uuid: "40f53301cb6f4b8caa91705d704132ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" is_pre_election: true
I20260806 02:40:20.974505 30759 leader_election.cc:302] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f53301cb6f4b8caa91705d704132ad, 4ec881b239b7446488480924acc29572; no voters:
I20260806 02:40:20.974705 30650 raft_consensus.cc:2454] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f53301cb6f4b8caa91705d704132ad in term 0.
I20260806 02:40:20.974740 30954 raft_consensus.cc:2781] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:20.974823 30954 raft_consensus.cc:482] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:20.974874 30954 raft_consensus.cc:3037] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:20.975939 30954 raft_consensus.cc:504] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.976316 30954 leader_election.cc:288] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [CANDIDATE]: Term 1 election: Requested vote from peers 4ec881b239b7446488480924acc29572 (127.28.191.1:33013), fc7f126b881a4b9aa1fbb5d10c41fde9 (127.28.191.2:46579)
I20260806 02:40:20.976653 30650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06a181fea094f0ba649fbe9b8fbed91" candidate_uuid: "40f53301cb6f4b8caa91705d704132ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9"
I20260806 02:40:20.976644 30470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06a181fea094f0ba649fbe9b8fbed91" candidate_uuid: "40f53301cb6f4b8caa91705d704132ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec881b239b7446488480924acc29572"
I20260806 02:40:20.976794 30470 raft_consensus.cc:3037] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:20.976794 30650 raft_consensus.cc:3037] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:20.978117 30650 raft_consensus.cc:2454] T d06a181fea094f0ba649fbe9b8fbed91 P fc7f126b881a4b9aa1fbb5d10c41fde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f53301cb6f4b8caa91705d704132ad in term 1.
I20260806 02:40:20.978117 30470 raft_consensus.cc:2454] T d06a181fea094f0ba649fbe9b8fbed91 P 4ec881b239b7446488480924acc29572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f53301cb6f4b8caa91705d704132ad in term 1.
I20260806 02:40:20.978415 30760 leader_election.cc:302] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f53301cb6f4b8caa91705d704132ad, fc7f126b881a4b9aa1fbb5d10c41fde9; no voters:
I20260806 02:40:20.978608 30954 raft_consensus.cc:2781] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:20.978827 30954 raft_consensus.cc:686] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [term 1 LEADER]: Becoming Leader. State: Replica: 40f53301cb6f4b8caa91705d704132ad, State: Running, Role: LEADER
I20260806 02:40:20.979132 30954 consensus_queue.cc:237] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } }
I20260806 02:40:20.982354 30256 catalog_manager.cc:5495] T d06a181fea094f0ba649fbe9b8fbed91 P 40f53301cb6f4b8caa91705d704132ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 40f53301cb6f4b8caa91705d704132ad (127.28.191.3). New cstate: current_term: 1 leader_uuid: "40f53301cb6f4b8caa91705d704132ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec881b239b7446488480924acc29572" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 33013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc7f126b881a4b9aa1fbb5d10c41fde9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40f53301cb6f4b8caa91705d704132ad" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 43109 } health_report { overall_health: HEALTHY } } }
Aug 06 02:40:20 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:20.999769 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:20.991039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.28.191.1:33013)
0806 02:40:20.991204 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:20.991207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:20.991290 (+ 83us) server_negotiation.cc:407] Connection header received
0806 02:40:20.991406 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:20.991408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:20.991449 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:20.991510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:20.992195 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.992674 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.993565 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:20.993785 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:20.993868 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:20.996314 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:20.996327 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:20.996342 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:20.996366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:20.998162 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.998594 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.998599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.998601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.998687 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:20.998909 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:20.998911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:20.998912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:20.999149 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:20.999314 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:20.999440 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:40:20.999537 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:21.028436 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:21.020067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.28.191.1:33013)
0806 02:40:21.020191 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:21.020195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:21.020353 (+ 158us) server_negotiation.cc:407] Connection header received
0806 02:40:21.020523 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:21.020526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:21.020568 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:21.020645 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:21.021392 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:21.021915 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:21.022724 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:21.022951 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:21.023060 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:21.025339 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:21.025343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:21.025344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:21.025366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:21.026793 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:21.027189 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:21.027193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:21.027195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:21.027272 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:21.027529 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:21.027533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:21.027535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:21.027692 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:21.027820 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:21.028109 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 02:40:21.028256 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:21.053151 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:21.045612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.28.191.1:33013)
0806 02:40:21.045764 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:21.045768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:21.045891 (+ 123us) server_negotiation.cc:407] Connection header received
0806 02:40:21.046063 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:21.046070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:21.046107 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:21.046189 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:21.046841 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:21.047324 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:21.048130 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:21.048335 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:21.048431 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:21.050606 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:21.050610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:21.050612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:21.050633 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:21.051813 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:21.052181 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:21.052184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:21.052185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:21.052252 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:21.052472 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:21.052474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:21.052475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:21.052616 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:21.052707 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:21.052890 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 02:40:21.052970 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:40:21.054709 30416 tablet_service.cc:2908] Rejecting scan request for tablet d06a181fea094f0ba649fbe9b8fbed91: Uninitialized: clean time has not yet been initialized
I20260806 02:40:21.056758 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 30348
I20260806 02:40:21.068583 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 30543
I20260806 02:40:21.076066 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 30741
I20260806 02:40:21.090471 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 30205
2026-08-06T02:40:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2461 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30979](info): set up 2 sockets
Aug 06 02:40:21 dist-test-slave-tfpf krb5kdc[30979](info): commencing operation
krb5kdc: starting...
W20260806 02:40:22.273445 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:22.389489 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43879
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:22.494807 30997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:22.495097 30997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:22.495177 30997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:22.499130 30997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:22.499207 30997 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:40:22.499235 30997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:22.499272 30997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:22.499310 30997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:22.499341 30997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:22.504434 30997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:43879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:22.505405 30997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:22.506333 30997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:22.514377 31006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:22.514428 30997 server_base.cc:1034] running on GCE node
W20260806 02:40:22.514384 31003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:22.514577 31004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:22.515110 30997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:22.516083 30997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:22.517251 30997 hybrid_clock.cc:648] HybridClock initialized: now 1785984022517228 us; error 35 us; skew 500 ppm
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:22.524750 30997 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:22.525635 30997 webserver.cc:460] Webserver started at http://127.28.191.62:35995/ using document root <none> and password file <none>
I20260806 02:40:22.526113 30997 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:22.526170 30997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:22.526320 30997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:22.528223 30997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "d6ed7d150eb34b5da06eba8b447ea4b1"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "4c2f2823ae3dc699ba16ff0b06c9a7c1"
server_key_iv: "68be361974244161ebf76025e8705845"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.528728 30997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "d6ed7d150eb34b5da06eba8b447ea4b1"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "4c2f2823ae3dc699ba16ff0b06c9a7c1"
server_key_iv: "68be361974244161ebf76025e8705845"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.532171 30997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:22.534374 31017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:22.535483 30997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:22.535609 30997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "d6ed7d150eb34b5da06eba8b447ea4b1"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "4c2f2823ae3dc699ba16ff0b06c9a7c1"
server_key_iv: "68be361974244161ebf76025e8705845"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.535717 30997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:22.550673 30997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:22.553597 30997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:22.553782 30997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:22.562115 30997 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:43879
I20260806 02:40:22.562124 31100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:43879 every 8 connection(s)
I20260806 02:40:22.562999 30997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:22.565559 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 30997
I20260806 02:40:22.565714 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:22.565698 31101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:22.565982 29436 external_mini_cluster.cc:1428] Setting key 660502098417ecb3903cd5212ce38deb
I20260806 02:40:22.572574 31101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Bootstrap starting.
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:22.574929 31101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:22.575868 31101 log.cc:824] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:22.578167 31101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: No bootstrap required, opened a new log
I20260806 02:40:22.580545 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:22.567262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.28.191.62:43879)
0806 02:40:22.567675 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:22.567694 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:22.567735 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:40:22.568385 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:22.568407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:22.568802 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:22.569150 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:22.569769 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.571158 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.571945 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.572653 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.572822 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:22.575239 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:22.575245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:22.575282 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:22.575325 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:22.577888 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.578368 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.578373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.578377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.578487 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.578790 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.578795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.578796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.579160 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:22.579347 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:22.579649 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:40:22.579907 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260806 02:40:22.581187 31101 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } }
I20260806 02:40:22.581415 31101 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:22.581496 31101 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ed7d150eb34b5da06eba8b447ea4b1, State: Initialized, Role: FOLLOWER
I20260806 02:40:22.581979 31101 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } }
I20260806 02:40:22.582158 31101 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:22.582222 31101 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:22.582312 31101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:22.583302 31101 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } }
I20260806 02:40:22.583611 31101 leader_election.cc:302] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ed7d150eb34b5da06eba8b447ea4b1; no voters:
I20260806 02:40:22.583839 31101 leader_election.cc:288] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:22.583972 31108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:22.584187 31108 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [term 1 LEADER]: Becoming Leader. State: Replica: d6ed7d150eb34b5da06eba8b447ea4b1, State: Running, Role: LEADER
I20260806 02:40:22.584514 31108 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } }
I20260806 02:40:22.585289 31101 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:22.586275 31109 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } } }
I20260806 02:40:22.586439 31109 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:22.586890 31118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:22.587592 31110 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ed7d150eb34b5da06eba8b447ea4b1. Latest consensus state: current_term: 1 leader_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ed7d150eb34b5da06eba8b447ea4b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43879 } } }
I20260806 02:40:22.587709 31110 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:22.589915 31118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:22.594877 31118 catalog_manager.cc:1347] Generated new cluster ID: b8d14aab7cf34b17bd67a22e3dccc567
I20260806 02:40:22.594949 31118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:22.600551 31118 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:22.601758 31118 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:22.614691 31118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Generated new TSK 0
I20260806 02:40:22.615409 31118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:22.647902 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43879
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:22.756731 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:22.756963 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:22.757031 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:22.760638 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:22.760685 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:22.760746 31135 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:22.760833 31135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:22.765787 31135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:22.766606 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:22.767488 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:22.776451 31142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:22.776492 31141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:22.776674 31135 server_base.cc:1034] running on GCE node
W20260806 02:40:22.776814 31145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:22.777247 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:22.777864 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:22.779037 31135 hybrid_clock.cc:648] HybridClock initialized: now 1785984022779005 us; error 44 us; skew 500 ppm
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:22.781684 31135 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:22.782608 31135 webserver.cc:460] Webserver started at http://127.28.191.1:33305/ using document root <none> and password file <none>
I20260806 02:40:22.783171 31135 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:22.783257 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:22.783452 31135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:22.785508 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "28a1293014e44ac9ba3bfd52e9b854fe"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "8fdf87db4a9b805fd7ede06397864e1c"
server_key_iv: "8985109c3f757c10d911d92797324f77"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.786091 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "28a1293014e44ac9ba3bfd52e9b854fe"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "8fdf87db4a9b805fd7ede06397864e1c"
server_key_iv: "8985109c3f757c10d911d92797324f77"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.789644 31135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:40:22.791899 31154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:22.793001 31135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 02:40:22.793161 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "28a1293014e44ac9ba3bfd52e9b854fe"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "8fdf87db4a9b805fd7ede06397864e1c"
server_key_iv: "8985109c3f757c10d911d92797324f77"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.793262 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:22.804484 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:22.807505 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:22.807693 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:22.809015 31135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:22.809141 31135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:22.809223 31135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:22.809268 31135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:22.820335 31135 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:39891
I20260806 02:40:22.820398 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:39891 every 8 connection(s)
I20260806 02:40:22.821429 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:22.822917 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:22.809927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:43079 (local address 127.28.191.62:43879)
0806 02:40:22.810053 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:22.810057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:22.811086 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 02:40:22.812171 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:22.812175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:22.812242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:22.812301 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:22.813961 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.814686 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.815816 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.816118 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.816227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:22.819106 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:22.819110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:22.819114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:22.819142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:22.820763 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.821340 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.821347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.821351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.821447 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.821812 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.821819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.821823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.822058 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:22.822175 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:22.822717 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:40:22.822811 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:40:22.823562 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 31135
I20260806 02:40:22.823652 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:22.823680 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:22.810443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.1:43079)
0806 02:40:22.810939 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:22.810979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:22.811900 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:22.812780 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:22.812788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:22.813234 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:22.813824 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:22.813840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.814810 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.814813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:22.815695 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:22.815705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.816283 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.816288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:22.816483 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:22.817179 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:22.817203 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:22.818945 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:22.820890 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:22.820894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:22.820904 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:22.821244 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:22.821547 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:22.821552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:22.821557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:22.821696 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:22.822228 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:22.822235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:22.822486 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 02:40:22.822630 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:40:22.822845 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260806 02:40:22.823957 29436 external_mini_cluster.cc:1428] Setting key a5f5adf160b1aa75fdc7ca49bdac6436
I20260806 02:40:22.825148 31322 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43879
I20260806 02:40:22.825316 31322 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:22.825835 31322 heartbeater.cc:511] Master 127.28.191.62:43879 requested a full tablet report, sending...
I20260806 02:40:22.827539 31040 ts_manager.cc:194] Registered new tserver with Master: 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891)
I20260806 02:40:22.828688 31040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:43079
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:22.836752 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:22.827242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.1:34827)
0806 02:40:22.827552 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:22.827569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:22.827668 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:22.828030 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:22.828035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:22.828252 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:22.829149 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:22.829160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.830189 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.830192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:22.830820 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:22.830830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.831370 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.831374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:22.831475 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:22.832200 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:22.832225 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:22.832702 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:22.834866 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:22.834873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:22.834876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:22.835169 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:22.835611 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:22.835614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:22.835617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:22.835780 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:22.836236 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:22.836237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:22.836303 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:40:22.836407 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:40:22.836548 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":128,"threads_started":1}
I20260806 02:40:22.836797 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:22.827283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:34827 (local address 127.28.191.62:43879)
0806 02:40:22.827582 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:22.827589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:22.827626 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:40:22.827760 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:22.827765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:22.827818 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:22.827905 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:22.829263 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.830054 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.830956 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:22.831254 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:22.831364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:22.832843 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:22.832847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:22.832884 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:22.832911 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:22.834721 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.835313 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.835319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.835320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.835425 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:22.835913 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:22.835919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:22.835924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:22.836117 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:22.836175 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:22.836524 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:40:22.836641 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:22.855921 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43879
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:22.965050 31340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:22.965281 31340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:22.965320 31340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:22.972144 31340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:22.972229 31340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:22.972323 31340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:22.972391 31340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:22.978201 31340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:22.979103 31340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:22.979981 31340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:22.989573 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:22.989615 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:22.989583 31348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:22.989835 31340 server_base.cc:1034] running on GCE node
I20260806 02:40:22.990296 31340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:22.990926 31340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:22.992122 31340 hybrid_clock.cc:648] HybridClock initialized: now 1785984022992107 us; error 27 us; skew 500 ppm
Aug 06 02:40:22 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:22.994900 31340 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:22.995772 31340 webserver.cc:460] Webserver started at http://127.28.191.2:45129/ using document root <none> and password file <none>
I20260806 02:40:22.996342 31340 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:22.996413 31340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:22.996574 31340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:22.998677 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "2adaaa7e89934b5280d62bc2f045bb95"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "584f141fd1794d289b7b297dd308fdf5"
server_key_iv: "0c1ada77f97ebd6659b5f2a1e2acdceb"
server_key_version: "encryptionkey@0"
I20260806 02:40:22.999181 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "2adaaa7e89934b5280d62bc2f045bb95"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "584f141fd1794d289b7b297dd308fdf5"
server_key_iv: "0c1ada77f97ebd6659b5f2a1e2acdceb"
server_key_version: "encryptionkey@0"
I20260806 02:40:23.002696 31340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:23.004966 31368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:23.006170 31340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:23.006287 31340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "2adaaa7e89934b5280d62bc2f045bb95"
format_stamp: "Formatted at 2026-08-06 02:40:22 on dist-test-slave-tfpf"
server_key: "584f141fd1794d289b7b297dd308fdf5"
server_key_iv: "0c1ada77f97ebd6659b5f2a1e2acdceb"
server_key_version: "encryptionkey@0"
I20260806 02:40:23.006384 31340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:23.016156 31340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:23.018872 31340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:23.019052 31340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:23.020329 31340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:23.020375 31340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:23.020435 31340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:23.020470 31340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:23.030812 31340 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:44373
I20260806 02:40:23.030913 31533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:44373 every 8 connection(s)
I20260806 02:40:23.031953 31340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:23.034494 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.021111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:50089 (local address 127.28.191.62:43879)
0806 02:40:23.021250 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.021254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.022102 (+ 848us) server_negotiation.cc:407] Connection header received
0806 02:40:23.023047 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.023050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.023109 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.023189 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.024864 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.025775 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.027640 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.027984 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.028071 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.030966 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.030970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.030973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.031001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.032587 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.033057 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.033063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.033085 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.033185 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.033519 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.033521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.033522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.033699 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.033776 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.034282 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.034382 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:40:23.035229 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.021501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.2:50089)
0806 02:40:23.021932 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.021980 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.022838 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.023475 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.023487 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.024004 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:23.024687 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.024710 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.025945 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.025967 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.027495 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.027510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.028130 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.028136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.028343 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.028910 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:23.028936 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:23.030736 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:23.032688 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.032696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.032711 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.032955 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.033275 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.033277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.033279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.033421 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.033785 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:23.033789 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:23.034077 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.034238 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.034460 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260806 02:40:23.036320 31536 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43879
I20260806 02:40:23.036468 31536 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:23.037006 31536 heartbeater.cc:511] Master 127.28.191.62:43879 requested a full tablet report, sending...
I20260806 02:40:23.038102 31041 ts_manager.cc:194] Registered new tserver with Master: 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2:44373)
I20260806 02:40:23.038844 31041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:50089
I20260806 02:40:23.042186 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 31340
I20260806 02:40:23.042311 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:23.042579 29436 external_mini_cluster.cc:1428] Setting key 72653e35fb536702b1510357f922d7df
I20260806 02:40:23.045931 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.038571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.2:37167)
0806 02:40:23.038984 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.038998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.039153 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.039384 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.039387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.039577 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:23.039672 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.039677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.040652 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.040655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.041272 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.041281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.041660 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.041663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.041727 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.042222 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:23.042232 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:23.042572 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:23.044495 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.044498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.044499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.044681 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.044962 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.044964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.044965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.045035 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.045569 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:23.045571 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:23.045641 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.045719 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.045813 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260806 02:40:23.046165 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.038629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:37167 (local address 127.28.191.62:43879)
0806 02:40:23.038755 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.038788 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.039087 (+ 299us) server_negotiation.cc:407] Connection header received
0806 02:40:23.039246 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.039248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.039285 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.039346 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.039822 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.040536 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.041367 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.041569 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.041647 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.042732 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.042736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.042738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.042763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.044397 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.044774 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.044780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.044784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.044875 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.045177 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.045183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.045187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.045478 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.045584 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.045801 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.045929 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:23.071918 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43879
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:23.181221 31551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:23.181447 31551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:23.181515 31551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:23.185631 31551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:23.185706 31551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:23.185774 31551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:23.185858 31551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:23.190999 31551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:23.191972 31551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:23.192929 31551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:23.202260 31560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:23.202283 31556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:23.202266 31557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:23.202386 31551 server_base.cc:1034] running on GCE node
I20260806 02:40:23.202924 31551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:23.203438 31551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:23.204619 31551 hybrid_clock.cc:648] HybridClock initialized: now 1785984023204600 us; error 37 us; skew 500 ppm
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:23.207195 31551 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:23.207968 31551 webserver.cc:460] Webserver started at http://127.28.191.3:32911/ using document root <none> and password file <none>
I20260806 02:40:23.208496 31551 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:23.208580 31551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:23.208782 31551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:23.210803 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
format_stamp: "Formatted at 2026-08-06 02:40:23 on dist-test-slave-tfpf"
server_key: "39fbed308c4e216b14e0dc45d52cf6fd"
server_key_iv: "2a25390697823196443e95010737dcb6"
server_key_version: "encryptionkey@0"
I20260806 02:40:23.211412 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
format_stamp: "Formatted at 2026-08-06 02:40:23 on dist-test-slave-tfpf"
server_key: "39fbed308c4e216b14e0dc45d52cf6fd"
server_key_iv: "2a25390697823196443e95010737dcb6"
server_key_version: "encryptionkey@0"
I20260806 02:40:23.214864 31551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:23.217469 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:23.218648 31551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:23.218880 31551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
format_stamp: "Formatted at 2026-08-06 02:40:23 on dist-test-slave-tfpf"
server_key: "39fbed308c4e216b14e0dc45d52cf6fd"
server_key_iv: "2a25390697823196443e95010737dcb6"
server_key_version: "encryptionkey@0"
I20260806 02:40:23.218981 31551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:23.227715 31551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:23.230242 31551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:23.230422 31551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:23.231719 31551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:23.231788 31551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:23.231858 31551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:23.231896 31551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:23.242290 31551 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:39441
I20260806 02:40:23.242328 31732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:39441 every 8 connection(s)
I20260806 02:40:23.243167 31551 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:23.245561 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.232470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:36545 (local address 127.28.191.62:43879)
0806 02:40:23.232608 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.232613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.233402 (+ 789us) server_negotiation.cc:407] Connection header received
0806 02:40:23.234261 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.234265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.234322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.234404 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.235935 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.236817 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.238685 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.239032 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.239151 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.241847 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.241851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.241854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.241883 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.243530 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.244107 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.244110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.244112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.244194 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.244513 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.244516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.244518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.244692 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.244794 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.245290 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.245402 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:40:23.246204 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.232814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.3:36545)
0806 02:40:23.233256 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.233302 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.234074 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.234637 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.234648 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.235130 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:23.235763 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.235786 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.236960 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.236967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.238544 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.238560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.239185 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.239193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.239401 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.239960 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:23.239983 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:23.241693 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:23.243685 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.243691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.243703 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.243997 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.244280 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.244284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.244286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.244414 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.244793 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:23.244799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:23.245014 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.245230 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.245504 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260806 02:40:23.247210 31733 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43879
I20260806 02:40:23.247355 31733 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:23.247800 31733 heartbeater.cc:511] Master 127.28.191.62:43879 requested a full tablet report, sending...
I20260806 02:40:23.248085 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 31551
I20260806 02:40:23.248204 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:23.248493 29436 external_mini_cluster.cc:1428] Setting key 13d1c71aa6640b413ecaf66fff06dcd7
I20260806 02:40:23.248941 31040 ts_manager.cc:194] Registered new tserver with Master: 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441)
I20260806 02:40:23.249711 31040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:36545
I20260806 02:40:23.250721 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:40:23.256808 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.249554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:51561 (local address 127.28.191.62:43879)
0806 02:40:23.249641 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.249644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.249913 (+ 269us) server_negotiation.cc:407] Connection header received
0806 02:40:23.250099 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.250104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.250149 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.250222 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.250722 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.251276 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.252104 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.252321 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.252436 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.253714 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.253717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.253718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.253739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.255121 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.255551 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.255557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.255561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.255648 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.255934 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.255938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.255940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.256119 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.256248 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.256527 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.256637 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 02:40:23.256819 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.249527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.28.191.3:51561)
0806 02:40:23.249828 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.249844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.250002 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.250258 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.250261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.250479 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:23.250603 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.250610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.251413 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.251416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.251977 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.251986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.252437 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.252441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.252539 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.253086 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:23.253108 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:23.253578 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:23.255238 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.255244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.255247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.255449 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.255740 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.255745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.255747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.255833 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.256237 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:23.256241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:23.256341 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.256458 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.256600 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:23.276527 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.269289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.28.191.62:43879)
0806 02:40:23.269413 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.269416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.269426 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:40:23.269457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.269459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.269492 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.269557 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.270229 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.270700 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.271491 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.271684 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.271755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.273921 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.273924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.273925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.273946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.275278 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.275620 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.275622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.275624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.275690 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.275909 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.275911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.275912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.276063 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.276157 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.276343 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.276425 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:40:23.279204 31041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:23.282120 31041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:23.295640 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.290117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.28.191.3:39441)
0806 02:40:23.290846 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.290865 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.290889 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:40:23.290980 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.290986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.291196 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.291359 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:23.291677 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.292616 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.294264 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.294970 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.295087 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.295253 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.295409 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.295503 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":118,"threads_started":1}
I20260806 02:40:23.295626 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.289978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:39441 (local address 127.0.0.1:33248)
0806 02:40:23.290383 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.290430 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.290555 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.291344 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.291350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.291374 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:23.291518 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.291525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.292748 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.292752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.294110 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.294121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.295087 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.295091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.295190 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.295225 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.295366 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.295496 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":80,"threads_started":1}
I20260806 02:40:23.296196 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.290077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39891 (local address 127.0.0.1:60154)
0806 02:40:23.291248 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.291262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.291354 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.291912 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.291915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.291933 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:23.292049 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.292056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.293495 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.293498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.294593 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.294602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.295789 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.295792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.295881 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.295904 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.295979 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.296053 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260806 02:40:23.296406 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.291136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.28.191.1:39891)
0806 02:40:23.291526 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.291537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.291559 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:40:23.291641 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.291646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.291787 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.291934 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:23.292207 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.293365 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.294730 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.295485 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.295592 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.295727 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.296056 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.296179 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":90,"threads_started":1}
I20260806 02:40:23.297405 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.290696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:44373 (local address 127.0.0.1:55596)
0806 02:40:23.292182 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.292195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.292295 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.292669 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.292672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.292689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:23.292807 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.292814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.294084 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.294088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.295069 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.295080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.296385 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.296395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.296927 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.296952 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.297159 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.297304 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260806 02:40:23.297860 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.290854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.28.191.2:44373)
0806 02:40:23.291612 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.291623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.292281 (+ 658us) server_negotiation.cc:407] Connection header received
0806 02:40:23.292403 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.292408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.292548 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.292690 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:23.292919 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.293968 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.295205 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.295959 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.296054 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.296215 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.297682 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.297756 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":126,"threads_started":1}
I20260806 02:40:23.299114 31638 tablet_service.cc:1467] Processing CreateTablet for tablet 7200c910d97d418d96c3781e6ce83088 (DEFAULT_TABLE table=test-table [id=e89bd881ce364cdca651c03cd60080f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.299117 31227 tablet_service.cc:1467] Processing CreateTablet for tablet 7200c910d97d418d96c3781e6ce83088 (DEFAULT_TABLE table=test-table [id=e89bd881ce364cdca651c03cd60080f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.299896 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 7200c910d97d418d96c3781e6ce83088 (DEFAULT_TABLE table=test-table [id=e89bd881ce364cdca651c03cd60080f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.300235 31227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7200c910d97d418d96c3781e6ce83088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.300240 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7200c910d97d418d96c3781e6ce83088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.300833 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7200c910d97d418d96c3781e6ce83088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.307334 31763 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Bootstrap starting.
I20260806 02:40:23.308954 31766 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Bootstrap starting.
I20260806 02:40:23.309825 31763 tablet_bootstrap.cc:654] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.310694 31763 log.cc:824] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:23.311012 31766 tablet_bootstrap.cc:654] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.311648 31765 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Bootstrap starting.
I20260806 02:40:23.311864 31766 log.cc:824] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:23.313119 31763 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: No bootstrap required, opened a new log
I20260806 02:40:23.313313 31763 ts_tablet_manager.cc:1397] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:40:23.313637 31765 tablet_bootstrap.cc:654] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.313864 31766 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: No bootstrap required, opened a new log
I20260806 02:40:23.314059 31766 ts_tablet_manager.cc:1397] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:40:23.314455 31765 log.cc:824] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:23.316401 31765 tablet_bootstrap.cc:492] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: No bootstrap required, opened a new log
I20260806 02:40:23.316594 31765 ts_tablet_manager.cc:1397] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 02:40:23.316690 31763 raft_consensus.cc:348] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.316964 31763 raft_consensus.cc:374] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.317019 31763 raft_consensus.cc:729] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adaaa7e89934b5280d62bc2f045bb95, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.317487 31763 consensus_queue.cc:260] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.317785 31766 raft_consensus.cc:348] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.318034 31766 raft_consensus.cc:374] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.318109 31766 raft_consensus.cc:729] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48db77592e4a4ad5ac2da1e16df61cc8, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.318483 31536 heartbeater.cc:503] Master 127.28.191.62:43879 was elected leader, sending a full tablet report...
I20260806 02:40:23.318532 31766 consensus_queue.cc:260] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.319139 31733 heartbeater.cc:503] Master 127.28.191.62:43879 was elected leader, sending a full tablet report...
I20260806 02:40:23.319473 31766 ts_tablet_manager.cc:1428] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:40:23.320467 31765 raft_consensus.cc:348] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.320688 31765 raft_consensus.cc:374] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.320752 31765 raft_consensus.cc:729] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a1293014e44ac9ba3bfd52e9b854fe, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.321198 31765 consensus_queue.cc:260] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.321599 31763 ts_tablet_manager.cc:1428] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:40:23.322168 31765 ts_tablet_manager.cc:1428] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:40:23.322453 31322 heartbeater.cc:503] Master 127.28.191.62:43879 was elected leader, sending a full tablet report...
W20260806 02:40:23.322835 31323 tablet.cc:2369] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:40:23.494161 31735 tablet.cc:2369] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:40:23.533313 31538 tablet.cc:2369] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:23.577219 31771 raft_consensus.cc:482] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:23.577414 31771 raft_consensus.cc:504] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.578480 31771 leader_election.cc:288] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441), 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891)
I20260806 02:40:23.582607 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.578828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:38025 (local address 127.28.191.1:39891)
0806 02:40:23.578954 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.578959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.579240 (+ 281us) server_negotiation.cc:407] Connection header received
0806 02:40:23.579388 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.579391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.579433 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.579538 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:23.579833 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.580332 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.581466 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.581890 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.581997 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.582039 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.582388 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.582478 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:40:23.582614 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.578725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39891 (local address 127.28.191.2:38025)
0806 02:40:23.579166 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.579177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.579262 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.579562 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.579566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.579601 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:23.579725 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.579730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.580481 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.580484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.581347 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.581356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.582121 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.582125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.582228 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.582285 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.582343 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.582426 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":97,"thread_start_us":84,"threads_started":1}
I20260806 02:40:23.582621 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.578873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:60985 (local address 127.28.191.3:39441)
0806 02:40:23.579026 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.579031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.579173 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:40:23.579362 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.579366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.579415 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.579471 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:23.579859 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.580328 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.581530 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.581909 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.582157 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.582208 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.582439 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.582518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:40:23.582648 31776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.578724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:39441 (local address 127.28.191.2:60985)
0806 02:40:23.579089 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.579102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.579226 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.579513 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.579519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.579540 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:23.579723 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.579730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.580459 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.580465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.581383 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.581393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.582160 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.582163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.582263 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.582305 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.582401 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.582469 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260806 02:40:23.583359 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7200c910d97d418d96c3781e6ce83088" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" is_pre_election: true
I20260806 02:40:23.583436 31252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7200c910d97d418d96c3781e6ce83088" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" is_pre_election: true
I20260806 02:40:23.583663 31252 raft_consensus.cc:2454] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 0.
I20260806 02:40:23.583663 31670 raft_consensus.cc:2454] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 0.
I20260806 02:40:23.584120 31369 leader_election.cc:302] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a1293014e44ac9ba3bfd52e9b854fe, 2adaaa7e89934b5280d62bc2f045bb95; no voters:
I20260806 02:40:23.584364 31771 raft_consensus.cc:2781] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:23.584442 31771 raft_consensus.cc:482] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:23.584484 31771 raft_consensus.cc:3037] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.585577 31771 raft_consensus.cc:504] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.586019 31771 leader_election.cc:288] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 election: Requested vote from peers 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441), 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891)
I20260806 02:40:23.586344 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7200c910d97d418d96c3781e6ce83088" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
I20260806 02:40:23.586367 31252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7200c910d97d418d96c3781e6ce83088" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a1293014e44ac9ba3bfd52e9b854fe"
I20260806 02:40:23.586484 31252 raft_consensus.cc:3037] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.586473 31670 raft_consensus.cc:3037] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.587519 31252 raft_consensus.cc:2454] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 1.
I20260806 02:40:23.587785 31369 leader_election.cc:302] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a1293014e44ac9ba3bfd52e9b854fe, 2adaaa7e89934b5280d62bc2f045bb95; no voters:
I20260806 02:40:23.587950 31771 raft_consensus.cc:2781] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:23.588109 31771 raft_consensus.cc:686] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 LEADER]: Becoming Leader. State: Replica: 2adaaa7e89934b5280d62bc2f045bb95, State: Running, Role: LEADER
I20260806 02:40:23.588107 31670 raft_consensus.cc:2454] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 1.
I20260806 02:40:23.588449 31771 consensus_queue.cc:237] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } }
I20260806 02:40:23.591429 31041 catalog_manager.cc:5495] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2). New cstate: current_term: 1 leader_uuid: "2adaaa7e89934b5280d62bc2f045bb95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } health_report { overall_health: HEALTHY } } }
I20260806 02:40:23.660550 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.657331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.28.191.2:44373)
0806 02:40:23.657475 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.657479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.657492 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:23.657649 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.657651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.657702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.657781 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:23.658052 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.658494 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.659336 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.659509 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.659575 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.659841 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:23.660196 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:23.660271 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.660356 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.660400 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:40:23.666409 31670 raft_consensus.cc:1261] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Refusing update from remote peer 2adaaa7e89934b5280d62bc2f045bb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:23.667140 31771 consensus_queue.cc:1035] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:23.668382 31252 raft_consensus.cc:1261] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Refusing update from remote peer 2adaaa7e89934b5280d62bc2f045bb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:23.669004 31771 consensus_queue.cc:1035] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:23.678750 31784 mvcc.cc:205] Tried to move back new op lower bound from 7315390560929374208 to 7315390560620900352. Current Snapshot: MvccSnapshot[applied={T|T < 7315390560929374208 or (T in {7315390560929374208})}]
I20260806 02:40:23.686107 31041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:23.686290 31041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:23.688884 31041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Sending DeleteTablet for 3 replicas of tablet 7200c910d97d418d96c3781e6ce83088
I20260806 02:40:23.689698 31638 tablet_service.cc:1514] Processing DeleteTablet for tablet 7200c910d97d418d96c3781e6ce83088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:33248
I20260806 02:40:23.689750 31227 tablet_service.cc:1514] Processing DeleteTablet for tablet 7200c910d97d418d96c3781e6ce83088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260806 02:40:23.690186 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 7200c910d97d418d96c3781e6ce83088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:55596
I20260806 02:40:23.690546 31795 tablet_replica.cc:330] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: stopping tablet replica
I20260806 02:40:23.690850 31795 raft_consensus.cc:2229] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:23.691242 31795 raft_consensus.cc:2258] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:23.691203 31041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:23.691499 31041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:23.692270 31794 tablet_replica.cc:330] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: stopping tablet replica
I20260806 02:40:23.692543 31794 raft_consensus.cc:2229] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:23.692875 31794 raft_consensus.cc:2258] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:23.693172 31795 ts_tablet_manager.cc:1905] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:23.694257 31794 ts_tablet_manager.cc:1905] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:23.696167 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 6168e8c77adc4f35aea595f440865d6b (DEFAULT_TABLE table=test-table [id=df1de2079a6e43d8955c304b36bb5da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.696259 31227 tablet_service.cc:1467] Processing CreateTablet for tablet 6168e8c77adc4f35aea595f440865d6b (DEFAULT_TABLE table=test-table [id=df1de2079a6e43d8955c304b36bb5da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.696388 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6168e8c77adc4f35aea595f440865d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.696514 31227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6168e8c77adc4f35aea595f440865d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.696518 31638 tablet_service.cc:1467] Processing CreateTablet for tablet 6168e8c77adc4f35aea595f440865d6b (DEFAULT_TABLE table=test-table [id=df1de2079a6e43d8955c304b36bb5da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:23.696759 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6168e8c77adc4f35aea595f440865d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.698616 31763 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Bootstrap starting.
I20260806 02:40:23.698997 31765 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Bootstrap starting.
I20260806 02:40:23.699016 31766 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Bootstrap starting.
I20260806 02:40:23.699479 31763 tablet_bootstrap.cc:654] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.699841 31765 tablet_bootstrap.cc:654] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.699841 31766 tablet_bootstrap.cc:654] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.700316 31795 ts_tablet_manager.cc:1918] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:23.700554 31795 log.cc:1196] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/7200c910d97d418d96c3781e6ce83088
I20260806 02:40:23.700919 31795 ts_tablet_manager.cc:1939] T 7200c910d97d418d96c3781e6ce83088 P 2adaaa7e89934b5280d62bc2f045bb95: Deleting consensus metadata
I20260806 02:40:23.701469 31794 ts_tablet_manager.cc:1918] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:23.701571 31794 log.cc:1196] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/7200c910d97d418d96c3781e6ce83088
I20260806 02:40:23.701918 31794 ts_tablet_manager.cc:1939] T 7200c910d97d418d96c3781e6ce83088 P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting consensus metadata
I20260806 02:40:23.702243 31018 catalog_manager.cc:4841] TS 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2:44373): tablet 7200c910d97d418d96c3781e6ce83088 (table test-table [id=e89bd881ce364cdca651c03cd60080f2]) successfully deleted
I20260806 02:40:23.702386 31763 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: No bootstrap required, opened a new log
I20260806 02:40:23.702472 31763 ts_tablet_manager.cc:1397] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 02:40:23.702919 31763 raft_consensus.cc:348] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.703056 31763 raft_consensus.cc:374] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.703042 31019 catalog_manager.cc:4841] TS 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441): tablet 7200c910d97d418d96c3781e6ce83088 (table test-table [id=e89bd881ce364cdca651c03cd60080f2]) successfully deleted
I20260806 02:40:23.703109 31763 raft_consensus.cc:729] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adaaa7e89934b5280d62bc2f045bb95, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.703230 31763 consensus_queue.cc:260] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.703308 31793 tablet_replica.cc:330] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: stopping tablet replica
I20260806 02:40:23.703471 31763 ts_tablet_manager.cc:1428] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:23.703583 31793 raft_consensus.cc:2229] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:23.703873 31793 raft_consensus.cc:2258] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:23.704838 31765 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: No bootstrap required, opened a new log
I20260806 02:40:23.704867 31766 tablet_bootstrap.cc:492] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: No bootstrap required, opened a new log
I20260806 02:40:23.704921 31765 ts_tablet_manager.cc:1397] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 02:40:23.704927 31766 ts_tablet_manager.cc:1397] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 02:40:23.705338 31766 raft_consensus.cc:348] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.705338 31765 raft_consensus.cc:348] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.705459 31765 raft_consensus.cc:374] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.705459 31766 raft_consensus.cc:374] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.705493 31766 raft_consensus.cc:729] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48db77592e4a4ad5ac2da1e16df61cc8, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.705493 31765 raft_consensus.cc:729] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a1293014e44ac9ba3bfd52e9b854fe, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.705603 31765 consensus_queue.cc:260] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.705608 31766 consensus_queue.cc:260] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.705842 31765 ts_tablet_manager.cc:1428] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:23.705842 31766 ts_tablet_manager.cc:1428] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:23.707024 31793 ts_tablet_manager.cc:1905] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:23.709830 31793 ts_tablet_manager.cc:1918] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:23.709916 31793 log.cc:1196] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/7200c910d97d418d96c3781e6ce83088
I20260806 02:40:23.710192 31793 ts_tablet_manager.cc:1939] T 7200c910d97d418d96c3781e6ce83088 P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting consensus metadata
I20260806 02:40:23.711196 31018 catalog_manager.cc:4841] TS 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891): tablet 7200c910d97d418d96c3781e6ce83088 (table test-table [id=e89bd881ce364cdca651c03cd60080f2]) successfully deleted
I20260806 02:40:23.803084 31771 raft_consensus.cc:482] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:23.803231 31771 raft_consensus.cc:504] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.803658 31771 leader_election.cc:288] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891), 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441)
I20260806 02:40:23.804069 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6168e8c77adc4f35aea595f440865d6b" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" is_pre_election: true
I20260806 02:40:23.804075 31252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6168e8c77adc4f35aea595f440865d6b" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" is_pre_election: true
I20260806 02:40:23.804234 31670 raft_consensus.cc:2454] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 0.
I20260806 02:40:23.804239 31252 raft_consensus.cc:2454] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 0.
I20260806 02:40:23.804525 31370 leader_election.cc:302] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adaaa7e89934b5280d62bc2f045bb95, 48db77592e4a4ad5ac2da1e16df61cc8; no voters:
I20260806 02:40:23.804706 31771 raft_consensus.cc:2781] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:23.804764 31771 raft_consensus.cc:482] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:23.804827 31771 raft_consensus.cc:3037] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.805848 31771 raft_consensus.cc:504] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.806183 31771 leader_election.cc:288] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 election: Requested vote from peers 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891), 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441)
I20260806 02:40:23.806491 31252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6168e8c77adc4f35aea595f440865d6b" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a1293014e44ac9ba3bfd52e9b854fe"
I20260806 02:40:23.806521 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6168e8c77adc4f35aea595f440865d6b" candidate_uuid: "2adaaa7e89934b5280d62bc2f045bb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
I20260806 02:40:23.806603 31670 raft_consensus.cc:3037] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.806610 31252 raft_consensus.cc:3037] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:23.807543 31670 raft_consensus.cc:2454] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 1.
I20260806 02:40:23.807791 31370 leader_election.cc:302] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adaaa7e89934b5280d62bc2f045bb95, 48db77592e4a4ad5ac2da1e16df61cc8; no voters:
I20260806 02:40:23.807801 31252 raft_consensus.cc:2454] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adaaa7e89934b5280d62bc2f045bb95 in term 1.
I20260806 02:40:23.807963 31771 raft_consensus.cc:2781] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:23.808063 31771 raft_consensus.cc:686] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 LEADER]: Becoming Leader. State: Replica: 2adaaa7e89934b5280d62bc2f045bb95, State: Running, Role: LEADER
I20260806 02:40:23.808211 31771 consensus_queue.cc:237] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.809934 31040 catalog_manager.cc:5495] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2). New cstate: current_term: 1 leader_uuid: "2adaaa7e89934b5280d62bc2f045bb95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:40:23 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:23.854920 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.846651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43879 (local address 127.0.0.1:46020)
0806 02:40:23.846981 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:23.846996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:23.847139 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:23.847414 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:23.847417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:23.847853 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:23.847948 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.847956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.848762 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.848768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.849320 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:23.849328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.849751 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.849754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:23.849822 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.850349 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:23.850361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:23.851749 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:23.853312 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.853323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.853329 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.853550 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.853852 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:23.853854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:23.853856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:23.853931 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:23.854291 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:23.854299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:23.854401 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 02:40:23.854544 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:40:23.854709 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
I20260806 02:40:23.854920 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:23.846724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.28.191.62:43879)
0806 02:40:23.846960 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:23.846969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:23.847082 (+ 113us) server_negotiation.cc:407] Connection header received
0806 02:40:23.847242 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:23.847247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:23.847289 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:23.847380 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:23.848092 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.848659 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.849436 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:23.849648 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:23.849751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:23.851883 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:23.851891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:23.851894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:23.851914 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:23.853219 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.853642 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.853647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.853649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.853758 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:23.854018 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:23.854021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:23.854023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:23.854190 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:23.854291 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:23.854593 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:40:23.854782 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260806 02:40:23.858752 31040 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:46020:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:40:23.859418 31040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:23.863751 31227 tablet_service.cc:1467] Processing CreateTablet for tablet c855414ee65e40a7883bd70f8b3276f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e29154f3f9034346bcbc412a3c31e39f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:23.863953 31638 tablet_service.cc:1467] Processing CreateTablet for tablet c855414ee65e40a7883bd70f8b3276f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e29154f3f9034346bcbc412a3c31e39f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:23.864171 31227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c855414ee65e40a7883bd70f8b3276f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.864257 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c855414ee65e40a7883bd70f8b3276f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.863751 31441 tablet_service.cc:1467] Processing CreateTablet for tablet c855414ee65e40a7883bd70f8b3276f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e29154f3f9034346bcbc412a3c31e39f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:23.866474 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c855414ee65e40a7883bd70f8b3276f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:23.866634 31766 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: Bootstrap starting.
I20260806 02:40:23.867653 31766 tablet_bootstrap.cc:654] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.868314 31765 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: Bootstrap starting.
I20260806 02:40:23.869021 31766 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: No bootstrap required, opened a new log
I20260806 02:40:23.869148 31766 ts_tablet_manager.cc:1397] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260806 02:40:23.869234 31765 tablet_bootstrap.cc:654] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.869619 31766 raft_consensus.cc:348] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.869731 31766 raft_consensus.cc:374] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.869730 31763 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: Bootstrap starting.
I20260806 02:40:23.869774 31766 raft_consensus.cc:729] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48db77592e4a4ad5ac2da1e16df61cc8, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.869899 31766 consensus_queue.cc:260] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.870167 31772 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:23.870324 31772 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:23.870177 31766 ts_tablet_manager.cc:1428] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:40:23.870389 31765 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: No bootstrap required, opened a new log
I20260806 02:40:23.870486 31765 ts_tablet_manager.cc:1397] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 02:40:23.870610 31763 tablet_bootstrap.cc:654] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:23.870957 31765 raft_consensus.cc:348] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.871091 31765 raft_consensus.cc:374] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.871138 31765 raft_consensus.cc:729] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a1293014e44ac9ba3bfd52e9b854fe, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.871264 31765 consensus_queue.cc:260] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.871529 31765 ts_tablet_manager.cc:1428] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:23.871752 31773 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:23.871883 31773 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:23.871930 31763 tablet_bootstrap.cc:492] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: No bootstrap required, opened a new log
I20260806 02:40:23.872015 31763 ts_tablet_manager.cc:1397] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:23.872417 31763 raft_consensus.cc:348] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.872552 31763 raft_consensus.cc:374] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:23.872599 31763 raft_consensus.cc:729] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adaaa7e89934b5280d62bc2f045bb95, State: Initialized, Role: FOLLOWER
I20260806 02:40:23.872733 31763 consensus_queue.cc:260] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:23.873018 31763 ts_tablet_manager.cc:1428] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:23.872962 31771 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:23.873116 31771 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:24.103965 31773 raft_consensus.cc:482] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:24.104094 31773 raft_consensus.cc:504] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:24.105243 31773 leader_election.cc:288] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2:44373), 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441)
I20260806 02:40:24.110333 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.105583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:48165 (local address 127.28.191.2:44373)
0806 02:40:24.105740 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.105756 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.105934 (+ 178us) server_negotiation.cc:407] Connection header received
0806 02:40:24.106021 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.106024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.106070 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.106153 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:24.106624 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.107427 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.109322 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.109755 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.109893 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.109936 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.110167 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.110229 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:40:24.110352 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.105455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:44373 (local address 127.28.191.1:48165)
0806 02:40:24.105847 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:24.105860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:24.105947 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:24.106175 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:24.106178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:24.106217 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:24.106512 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.106518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.107552 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.107556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.109202 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.109215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.109900 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.109904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.110005 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.110065 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:40:24.110143 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:40:24.110214 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":314,"spinlock_wait_cycles":29696,"thread_start_us":76,"threads_started":1}
I20260806 02:40:24.110773 31816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.105585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:42593 (local address 127.28.191.3:39441)
0806 02:40:24.105840 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.105843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.105855 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:24.106009 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.106013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.106070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.106154 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:24.106624 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.107424 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.109359 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.109798 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.109918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.109971 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.110270 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.110407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":58,"threads_started":1}
I20260806 02:40:24.110817 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.105454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:39441 (local address 127.28.191.1:42593)
0806 02:40:24.105782 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:24.105797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:24.105900 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:24.106175 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:24.106178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:24.106218 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:24.106508 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.106516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.107552 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.107556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.109203 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.109215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.110002 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.110005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.110125 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.110142 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:40:24.110235 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:40:24.110305 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":288,"spinlock_wait_cycles":4992,"thread_start_us":73,"threads_started":1}
I20260806 02:40:24.111027 31470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c855414ee65e40a7883bd70f8b3276f0" candidate_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adaaa7e89934b5280d62bc2f045bb95" is_pre_election: true
I20260806 02:40:24.111281 31470 raft_consensus.cc:2454] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a1293014e44ac9ba3bfd52e9b854fe in term 0.
I20260806 02:40:24.111325 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c855414ee65e40a7883bd70f8b3276f0" candidate_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" is_pre_election: true
I20260806 02:40:24.111472 31670 raft_consensus.cc:2454] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28a1293014e44ac9ba3bfd52e9b854fe in term 0.
I20260806 02:40:24.111703 31155 leader_election.cc:302] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a1293014e44ac9ba3bfd52e9b854fe, 2adaaa7e89934b5280d62bc2f045bb95; no voters:
I20260806 02:40:24.112000 31773 raft_consensus.cc:2781] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:24.112068 31773 raft_consensus.cc:482] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:24.112108 31773 raft_consensus.cc:3037] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:24.113219 31773 raft_consensus.cc:504] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:24.113526 31773 leader_election.cc:288] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [CANDIDATE]: Term 1 election: Requested vote from peers 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2:44373), 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441)
I20260806 02:40:24.113893 31670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c855414ee65e40a7883bd70f8b3276f0" candidate_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48db77592e4a4ad5ac2da1e16df61cc8"
I20260806 02:40:24.113893 31470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c855414ee65e40a7883bd70f8b3276f0" candidate_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adaaa7e89934b5280d62bc2f045bb95"
I20260806 02:40:24.114009 31670 raft_consensus.cc:3037] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:24.114015 31470 raft_consensus.cc:3037] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:24.115096 31670 raft_consensus.cc:2454] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a1293014e44ac9ba3bfd52e9b854fe in term 1.
I20260806 02:40:24.115096 31470 raft_consensus.cc:2454] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28a1293014e44ac9ba3bfd52e9b854fe in term 1.
I20260806 02:40:24.115358 31155 leader_election.cc:302] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a1293014e44ac9ba3bfd52e9b854fe, 2adaaa7e89934b5280d62bc2f045bb95; no voters:
I20260806 02:40:24.115520 31773 raft_consensus.cc:2781] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:24.115710 31773 raft_consensus.cc:686] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 LEADER]: Becoming Leader. State: Replica: 28a1293014e44ac9ba3bfd52e9b854fe, State: Running, Role: LEADER
I20260806 02:40:24.115943 31773 consensus_queue.cc:237] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } }
I20260806 02:40:24.117300 31817 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: TxnStatusTablet state changed. Reason: New leader 28a1293014e44ac9ba3bfd52e9b854fe. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.117405 31817 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:24.117727 31821 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:40:24.117937 31821 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:40:24.118518 31041 catalog_manager.cc:5495] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1). New cstate: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:24.164700 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.160629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.28.191.1:39891)
0806 02:40:24.160909 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.160918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.160936 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:40:24.161030 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.161033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.161174 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.161280 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:24.161543 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.162095 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.163307 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.163550 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.163630 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.163930 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.164313 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.164405 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.164523 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.164572 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":132,"threads_started":1}
I20260806 02:40:24.164714 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.160506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39891 (local address 127.0.0.1:60162)
0806 02:40:24.160739 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:24.160752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:24.160844 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:24.161311 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:24.161314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:24.161323 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:24.161432 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.161438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.162204 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.162207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.163156 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.163166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.163655 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.163659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.163749 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.163799 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.164412 (+ 613us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.164427 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 02:40:24.164497 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:40:24.164580 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260806 02:40:24.167935 31670 raft_consensus.cc:1261] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Refusing update from remote peer 28a1293014e44ac9ba3bfd52e9b854fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:24.168111 31470 raft_consensus.cc:1261] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Refusing update from remote peer 28a1293014e44ac9ba3bfd52e9b854fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:24.168491 31773 consensus_queue.cc:1035] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:24.168697 31817 consensus_queue.cc:1035] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:24.170087 31772 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: TxnStatusTablet state changed. Reason: New leader 28a1293014e44ac9ba3bfd52e9b854fe. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.170214 31772 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:24.170632 31771 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: TxnStatusTablet state changed. Reason: New leader 28a1293014e44ac9ba3bfd52e9b854fe. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.170737 31771 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:24.171890 31817 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.172003 31817 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:24.173291 31782 mvcc.cc:205] Tried to move back new op lower bound from 7315390562986663936 to 7315390562781208576. Current Snapshot: MvccSnapshot[applied={T|T < 7315390562986663936}]
I20260806 02:40:24.174144 31772 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.174265 31772 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 48db77592e4a4ad5ac2da1e16df61cc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:24.176623 31773 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.176761 31773 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 28a1293014e44ac9ba3bfd52e9b854fe: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:24.177706 31771 tablet_replica.cc:439] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 } } peers { permanent_uuid: "2adaaa7e89934b5280d62bc2f045bb95" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 44373 } } peers { permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 } } }
I20260806 02:40:24.177822 31771 tablet_replica.cc:442] T c855414ee65e40a7883bd70f8b3276f0 P 2adaaa7e89934b5280d62bc2f045bb95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:24.187876 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.184573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57119 (local address 127.28.191.1:39891)
0806 02:40:24.184785 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.184790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.184807 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:40:24.184854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.184857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.184912 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.185026 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:24.185314 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.185835 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.186796 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.187027 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.187143 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.187432 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.187514 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.187592 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.187692 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.187742 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:40:24.187907 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.184226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39891 (local address 127.28.191.2:57119)
0806 02:40:24.184551 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:24.184578 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:24.184720 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:24.185042 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:24.185045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:24.185054 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:24.185185 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.185192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.185948 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.185953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.186659 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.186687 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.187144 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.187147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.187236 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.187293 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.187609 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.187616 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:40:24.187683 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:40:24.187766 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260806 02:40:24.194383 31670 raft_consensus.cc:1261] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Refusing update from remote peer 2adaaa7e89934b5280d62bc2f045bb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:24.194423 31252 raft_consensus.cc:1261] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Refusing update from remote peer 2adaaa7e89934b5280d62bc2f045bb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:24.194741 31777 consensus_queue.cc:1035] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48db77592e4a4ad5ac2da1e16df61cc8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:24.194936 31771 consensus_queue.cc:1035] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a1293014e44ac9ba3bfd52e9b854fe" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:24.200547 31781 mvcc.cc:205] Tried to move back new op lower bound from 7315390563097034752 to 7315390561519656960. Current Snapshot: MvccSnapshot[applied={T|T < 7315390563097034752}]
I20260806 02:40:24.217159 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.212783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:45439 (local address 127.28.191.2:44373)
0806 02:40:24.212906 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.212911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.213247 (+ 336us) server_negotiation.cc:407] Connection header received
0806 02:40:24.213404 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.213407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.213455 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.213537 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:24.213793 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.214285 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.215908 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.216236 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.216356 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.216666 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.216758 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.216837 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.216942 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.217003 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:40:24.217228 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.212696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:44373 (local address 127.28.191.1:45439)
0806 02:40:24.213145 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:24.213159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:24.213297 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:24.213564 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:24.213567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:24.213576 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:24.213688 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.213693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.214408 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.214412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.215751 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:24.215763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.216354 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.216358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:24.216452 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.216512 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:24.216855 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:24.216863 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:40:24.216934 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:40:24.217039 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":84,"threads_started":1}
I20260806 02:40:24.241451 31041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:24.241639 31041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:24.243923 31041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6ed7d150eb34b5da06eba8b447ea4b1: Sending DeleteTablet for 3 replicas of tablet 6168e8c77adc4f35aea595f440865d6b
I20260806 02:40:24.244653 31638 tablet_service.cc:1514] Processing DeleteTablet for tablet 6168e8c77adc4f35aea595f440865d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:24 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:33248
I20260806 02:40:24.246812 31227 tablet_service.cc:1514] Processing DeleteTablet for tablet 6168e8c77adc4f35aea595f440865d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:24 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260806 02:40:24.247184 31846 tablet_replica.cc:330] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: stopping tablet replica
I20260806 02:40:24.247295 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 6168e8c77adc4f35aea595f440865d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:24 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:55596
I20260806 02:40:24.247367 31846 raft_consensus.cc:2229] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:24.247606 31846 raft_consensus.cc:2258] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:24.248448 31846 ts_tablet_manager.cc:1905] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:24.251791 31846 ts_tablet_manager.cc:1918] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:24.251941 31846 log.cc:1196] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/6168e8c77adc4f35aea595f440865d6b
I20260806 02:40:24.252363 31846 ts_tablet_manager.cc:1939] T 6168e8c77adc4f35aea595f440865d6b P 28a1293014e44ac9ba3bfd52e9b854fe: Deleting consensus metadata
Aug 06 02:40:24 dist-test-slave-tfpf krb5kdc[30979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:24.253149 31018 catalog_manager.cc:4841] TS 28a1293014e44ac9ba3bfd52e9b854fe (127.28.191.1:39891): tablet 6168e8c77adc4f35aea595f440865d6b (table test-table [id=df1de2079a6e43d8955c304b36bb5da8]) successfully deleted
I20260806 02:40:24.257233 31842 tablet_replica.cc:330] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: stopping tablet replica
I20260806 02:40:24.257223 31847 tablet_replica.cc:330] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: stopping tablet replica
I20260806 02:40:24.257349 31842 raft_consensus.cc:2229] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:24.257349 31847 raft_consensus.cc:2229] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:24.257469 31842 raft_consensus.cc:2258] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:24.257581 31847 raft_consensus.cc:2258] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:24.258257 31842 ts_tablet_manager.cc:1905] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:24.258281 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.246307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.28.191.1:39891)
0806 02:40:24.246434 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.246438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.246717 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:40:24.246902 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.246906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.246958 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.247059 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:24.248068 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.248868 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.250128 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.250446 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.250510 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.253631 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:24.253640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:24.253654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:24.253679 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:24.255731 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:24.256268 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:24.256275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:24.256277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:24.256389 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:24.256675 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:24.256678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:24.256680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:24.257009 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:24.257288 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.257913 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.258102 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 02:40:24.259514 31309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60164
I20260806 02:40:24.260241 31847 ts_tablet_manager.cc:1905] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:24.261188 31842 ts_tablet_manager.cc:1918] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:24.261260 31842 log.cc:1196] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/6168e8c77adc4f35aea595f440865d6b
I20260806 02:40:24.261525 31842 ts_tablet_manager.cc:1939] T 6168e8c77adc4f35aea595f440865d6b P 48db77592e4a4ad5ac2da1e16df61cc8: Deleting consensus metadata
I20260806 02:40:24.262308 31019 catalog_manager.cc:4841] TS 48db77592e4a4ad5ac2da1e16df61cc8 (127.28.191.3:39441): tablet 6168e8c77adc4f35aea595f440865d6b (table test-table [id=df1de2079a6e43d8955c304b36bb5da8]) successfully deleted
I20260806 02:40:24.262902 31847 ts_tablet_manager.cc:1918] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:24.262966 31847 log.cc:1196] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/6168e8c77adc4f35aea595f440865d6b
I20260806 02:40:24.263177 31847 ts_tablet_manager.cc:1939] T 6168e8c77adc4f35aea595f440865d6b P 2adaaa7e89934b5280d62bc2f045bb95: Deleting consensus metadata
I20260806 02:40:24.263739 31018 catalog_manager.cc:4841] TS 2adaaa7e89934b5280d62bc2f045bb95 (127.28.191.2:44373): tablet 6168e8c77adc4f35aea595f440865d6b (table test-table [id=df1de2079a6e43d8955c304b36bb5da8]) successfully deleted
I20260806 02:40:24.274514 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:24.265523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.28.191.62:43879)
0806 02:40:24.265648 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:24.265652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:24.265907 (+ 255us) server_negotiation.cc:407] Connection header received
0806 02:40:24.266030 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:24.266033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:24.266094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:24.266169 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:24.266923 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.267747 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.268963 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:24.269360 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:24.269458 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:24.271062 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:24.271065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:24.271067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:24.271091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:24.272623 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:24.273191 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:24.273195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:24.273196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:24.273280 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:24.273597 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:24.273601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:24.273602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:24.273791 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:24.273934 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:24.274191 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:40:24.274308 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 02:40:24.274902 31041 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46030
I20260806 02:40:24.276171 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 31135
I20260806 02:40:24.289865 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 31340
I20260806 02:40:24.303105 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 31551
I20260806 02:40:24.312325 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 30997
2026-08-06T02:40:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3222 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:24 dist-test-slave-tfpf krb5kdc[31855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:24 dist-test-slave-tfpf krb5kdc[31855](info): set up 2 sockets
Aug 06 02:40:24 dist-test-slave-tfpf krb5kdc[31855](info): commencing operation
krb5kdc: starting...
W20260806 02:40:25.508486 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:25 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:25.635782 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33351
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:25.745911 31870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:25.746208 31870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:25.746279 31870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:25.750079 31870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:25.750180 31870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:40:25.750213 31870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:25.750248 31870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:25.750275 31870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:25.750308 31870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:25.755407 31870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:33351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:25.756371 31870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:25.757431 31870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:25.765430 31882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:25.765525 31880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:25.765583 31870 server_base.cc:1034] running on GCE node
W20260806 02:40:25.765437 31879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:25.766166 31870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:25.767114 31870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:25.768273 31870 hybrid_clock.cc:648] HybridClock initialized: now 1785984025768246 us; error 36 us; skew 500 ppm
Aug 06 02:40:25 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:25.771070 31870 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:25.771916 31870 webserver.cc:460] Webserver started at http://127.28.191.62:41715/ using document root <none> and password file <none>
I20260806 02:40:25.772464 31870 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:25.772522 31870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:25.772775 31870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:25.774856 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "afd0f15865014998a864d84c58424f64"
format_stamp: "Formatted at 2026-08-06 02:40:25 on dist-test-slave-tfpf"
server_key: "eee60d47b1f12c79f353af47b912512a"
server_key_iv: "2c5a2d8100c44092c2f24eb7ff958b86"
server_key_version: "encryptionkey@0"
I20260806 02:40:25.775471 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "afd0f15865014998a864d84c58424f64"
format_stamp: "Formatted at 2026-08-06 02:40:25 on dist-test-slave-tfpf"
server_key: "eee60d47b1f12c79f353af47b912512a"
server_key_iv: "2c5a2d8100c44092c2f24eb7ff958b86"
server_key_version: "encryptionkey@0"
I20260806 02:40:25.779204 31870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:25.781561 31892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:25.782722 31870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:25.782868 31870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "afd0f15865014998a864d84c58424f64"
format_stamp: "Formatted at 2026-08-06 02:40:25 on dist-test-slave-tfpf"
server_key: "eee60d47b1f12c79f353af47b912512a"
server_key_iv: "2c5a2d8100c44092c2f24eb7ff958b86"
server_key_version: "encryptionkey@0"
I20260806 02:40:25.782984 31870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:25.795105 31870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:25.797977 31870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:25.798185 31870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:25.805794 31870 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:33351
I20260806 02:40:25.805913 31987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:33351 every 8 connection(s)
I20260806 02:40:25.806717 31870 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:25.809473 31988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:25.811694 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 31870
I20260806 02:40:25.811831 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:25.812106 29436 external_mini_cluster.cc:1428] Setting key c4cc276d9bdb0653d979856d93387b00
I20260806 02:40:25.815896 31988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Bootstrap starting.
I20260806 02:40:25.818614 31988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:25.819518 31988 log.cc:824] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:25.821738 31988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: No bootstrap required, opened a new log
Aug 06 02:40:25 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:25.826586 31988 raft_consensus.cc:348] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } }
I20260806 02:40:25.826891 31988 raft_consensus.cc:374] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:25.827000 31988 raft_consensus.cc:729] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd0f15865014998a864d84c58424f64, State: Initialized, Role: FOLLOWER
I20260806 02:40:25.827417 31988 consensus_queue.cc:260] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } }
I20260806 02:40:25.827556 31988 raft_consensus.cc:388] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:25.827647 31988 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:25.827669 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:25.813602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.28.191.62:33351)
0806 02:40:25.814029 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:25.814051 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:25.814095 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:40:25.814930 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:25.814954 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:25.815353 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:25.815714 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:25.816439 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:25.818018 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:25.818905 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:25.819605 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:25.819773 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:25.822492 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:25.822501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:25.822523 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:25.822566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:25.824623 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:25.825198 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:25.825203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:25.825207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:25.825319 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:25.825671 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:25.825673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:25.825674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:25.826036 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:25.826194 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:25.826547 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:40:25.826760 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
I20260806 02:40:25.827791 31988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:25.828722 31988 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } }
I20260806 02:40:25.829116 31988 leader_election.cc:302] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afd0f15865014998a864d84c58424f64; no voters:
I20260806 02:40:25.829444 31988 leader_election.cc:288] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:25.829553 31996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:25.829771 31996 raft_consensus.cc:686] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [term 1 LEADER]: Becoming Leader. State: Replica: afd0f15865014998a864d84c58424f64, State: Running, Role: LEADER
I20260806 02:40:25.830072 31996 consensus_queue.cc:237] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } }
I20260806 02:40:25.830670 31988 sys_catalog.cc:564] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:25.831552 31999 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afd0f15865014998a864d84c58424f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } } }
I20260806 02:40:25.831700 31999 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:25.832089 32010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:25.832055 32000 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd0f15865014998a864d84c58424f64. Latest consensus state: current_term: 1 leader_uuid: "afd0f15865014998a864d84c58424f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd0f15865014998a864d84c58424f64" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33351 } } }
I20260806 02:40:25.832165 32000 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:25.835055 32010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:25.840369 32010 catalog_manager.cc:1347] Generated new cluster ID: 26c2c1d5ccb74f0f9c5fdf5fe0af24f8
I20260806 02:40:25.840430 32010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:25.872545 32010 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:25.873936 32010 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:25.880713 32010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Generated new TSK 0
I20260806 02:40:25.881594 32010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:25.919994 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:26.030292 32033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:26.030530 32033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:26.030638 32033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:26.034772 32033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:26.034826 32033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:26.034860 32033 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:40:26.034904 32033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:26.034962 32033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:26.040038 32033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:26.041011 32033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:26.042037 32033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:26.051663 32039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:26.051623 32040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:26.051645 32043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:26.051983 32033 server_base.cc:1034] running on GCE node
I20260806 02:40:26.052424 32033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:26.053107 32033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:26.054282 32033 hybrid_clock.cc:648] HybridClock initialized: now 1785984026054253 us; error 39 us; skew 500 ppm
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:26.057113 32033 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:26.057899 32033 webserver.cc:460] Webserver started at http://127.28.191.1:37361/ using document root <none> and password file <none>
I20260806 02:40:26.058408 32033 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:26.058461 32033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:26.058619 32033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:26.060528 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "f2964edc64444e5e9599663671c844d6"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "0f7869bfb15b0e2fd0265101a69b39e5"
server_key_iv: "d81d3f0fef6ce5392b9f1da1cce89866"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.061020 32033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "f2964edc64444e5e9599663671c844d6"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "0f7869bfb15b0e2fd0265101a69b39e5"
server_key_iv: "d81d3f0fef6ce5392b9f1da1cce89866"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.064554 32033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:40:26.066820 32055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.067960 32033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:26.068095 32033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f2964edc64444e5e9599663671c844d6"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "0f7869bfb15b0e2fd0265101a69b39e5"
server_key_iv: "d81d3f0fef6ce5392b9f1da1cce89866"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.068199 32033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:26.083122 32033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:26.085913 32033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:26.086112 32033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:26.087512 32033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:26.087572 32033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.087630 32033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:26.087661 32033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:26.098074 32033 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:40623
I20260806 02:40:26.098150 32222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:40623 every 8 connection(s)
I20260806 02:40:26.099005 32033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:26.100934 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.088332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:58177 (local address 127.28.191.62:33351)
0806 02:40:26.088479 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.088484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.089362 (+ 878us) server_negotiation.cc:407] Connection header received
0806 02:40:26.090452 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.090459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.090536 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.090636 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.092158 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.093056 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.094412 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.094648 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.094764 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.097283 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.097286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.097289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.097316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.098904 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.099444 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.099449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.099451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.099550 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.099956 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.099960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.099961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.100197 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.100287 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.100708 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.100818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:40:26.101687 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.088688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.1:58177)
0806 02:40:26.089205 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.089252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.090214 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.090873 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.090885 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.091334 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.091966 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.091987 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.093198 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.093201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.094269 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.094279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.094809 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.094813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.094969 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.095492 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.095506 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.097111 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.099031 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.099037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.099057 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.099337 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.099660 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.099663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.099665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.099845 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.100303 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.100308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.100501 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.100654 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.100870 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":171,"threads_started":1}
I20260806 02:40:26.103003 32223 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33351
I20260806 02:40:26.103164 32223 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:26.103607 32223 heartbeater.cc:511] Master 127.28.191.62:33351 requested a full tablet report, sending...
I20260806 02:40:26.105113 31918 ts_manager.cc:194] Registered new tserver with Master: f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623)
I20260806 02:40:26.106168 31918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:58177
I20260806 02:40:26.106509 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 32033
I20260806 02:40:26.106643 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:26.106953 29436 external_mini_cluster.cc:1428] Setting key 255243959b712405fa0c7b2b8cb113cf
I20260806 02:40:26.112247 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.105021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:55775 (local address 127.28.191.62:33351)
0806 02:40:26.105197 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.105201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.105322 (+ 121us) server_negotiation.cc:407] Connection header received
0806 02:40:26.105421 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.105424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.105463 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.105553 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.105970 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.106553 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.107508 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.107816 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.107913 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.108971 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.108977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.108982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.109010 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.110493 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.110982 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.110985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.110985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.111059 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.111422 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.111427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.111430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.111651 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.111752 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.112057 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.112147 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 02:40:26.112419 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.104947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.1:55775)
0806 02:40:26.105224 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.105240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.105337 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.105572 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.105575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.105754 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.105844 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.105849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.106723 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.106726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.107384 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.107393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.107943 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.107946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.108033 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.108444 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.108458 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.108839 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.110624 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.110629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.110633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.110885 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.111151 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.111155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.111159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.111287 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.111761 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.111764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.111863 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.111991 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.112139 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:26.136436 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:26.241640 32235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:26.241858 32235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:26.241900 32235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:26.245617 32235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:26.245661 32235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:26.245707 32235 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:40:26.245720 32235 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:26.245747 32235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:26.250682 32235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:26.251638 32235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:26.252562 32235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:26.261950 32241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:26.262041 32243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:26.261950 32246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:26.262511 32235 server_base.cc:1034] running on GCE node
I20260806 02:40:26.262892 32235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:26.263491 32235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:26.264698 32235 hybrid_clock.cc:648] HybridClock initialized: now 1785984026264685 us; error 28 us; skew 500 ppm
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:26.267531 32235 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:26.268421 32235 webserver.cc:460] Webserver started at http://127.28.191.2:39899/ using document root <none> and password file <none>
I20260806 02:40:26.268982 32235 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:26.269126 32235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:26.269369 32235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:26.271324 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "0c8d456875a54034afc6616c80c5127e"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "b6d9044a7b11ad06885b0f0aad412a80"
server_key_iv: "8b794d93b7dd2544a465a96a7c3aee6a"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.272014 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "0c8d456875a54034afc6616c80c5127e"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "b6d9044a7b11ad06885b0f0aad412a80"
server_key_iv: "8b794d93b7dd2544a465a96a7c3aee6a"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.275614 32235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:26.277979 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.279132 32235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:26.279265 32235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "0c8d456875a54034afc6616c80c5127e"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "b6d9044a7b11ad06885b0f0aad412a80"
server_key_iv: "8b794d93b7dd2544a465a96a7c3aee6a"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.279384 32235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:26.299185 32235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:26.302716 32235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:26.302910 32235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:26.304455 32235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:26.304529 32235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.304597 32235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:26.304627 32235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:26.315424 32235 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:38117
I20260806 02:40:26.315480 32429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:38117 every 8 connection(s)
I20260806 02:40:26.316283 32235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:26.318455 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.305226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57575 (local address 127.28.191.62:33351)
0806 02:40:26.305368 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.305373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.306183 (+ 810us) server_negotiation.cc:407] Connection header received
0806 02:40:26.307091 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.307095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.307145 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.307202 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.308488 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.309252 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.310667 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.310910 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.311036 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.314735 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.314739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.314741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.314775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.316342 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.317030 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.317034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.317037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.317159 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.317491 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.317499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.317501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.317674 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.317754 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.318224 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.318334 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:40:26.319149 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.305598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.2:57575)
0806 02:40:26.306019 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.306066 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.306845 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.307594 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.307604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.307935 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.308343 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.308355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.309381 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.309385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.310505 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.310514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.311091 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.311095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.311262 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.311881 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.311908 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.314569 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.316472 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.316478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.316491 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.316890 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.317260 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.317262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.317264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.317391 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.317769 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.317773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.317986 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.318148 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.318381 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260806 02:40:26.320302 32430 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33351
I20260806 02:40:26.320427 32430 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:26.320905 32430 heartbeater.cc:511] Master 127.28.191.62:33351 requested a full tablet report, sending...
I20260806 02:40:26.322048 31920 ts_manager.cc:194] Registered new tserver with Master: 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:26.322415 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 32235
I20260806 02:40:26.322536 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:26.322753 31920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:57575
I20260806 02:40:26.322837 29436 external_mini_cluster.cc:1428] Setting key 9cf32e60513b872ca2712520876b00aa
I20260806 02:40:26.330595 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.322752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.2:59335)
0806 02:40:26.323125 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.323138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.323215 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.323448 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.323451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.323678 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.323779 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.323787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.324514 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.324517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.325371 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.325382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.325851 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.325856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.325959 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.326431 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.326453 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.326930 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.328846 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.328851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.328855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.329141 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.329425 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.329429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.329434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.329570 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.330101 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.330103 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.330183 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.330277 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.330379 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":75,"threads_started":1}
I20260806 02:40:26.330688 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.322738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:59335 (local address 127.28.191.62:33351)
0806 02:40:26.323241 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.323245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.323260 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:40:26.323295 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.323300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.323340 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.323416 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.323901 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.324377 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.325519 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.325714 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.325798 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.327120 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.327126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.327131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.327159 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.328730 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.329248 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.329251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.329252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.329318 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.329691 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.329697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.329701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.329984 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.330097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.330363 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.330485 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:26.355579 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:26.464200 32445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:26.464403 32445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:26.464444 32445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:26.468335 32445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:26.468389 32445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:26.468461 32445 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:40:26.468502 32445 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:26.468570 32445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:26.473798 32445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:26.474713 32445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:26.475656 32445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:26.485357 32453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:26.485381 32445 server_base.cc:1034] running on GCE node
W20260806 02:40:26.485345 32450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:26.485332 32456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:26.485982 32445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:26.486661 32445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:26.487818 32445 hybrid_clock.cc:648] HybridClock initialized: now 1785984026487805 us; error 25 us; skew 500 ppm
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:26.490758 32445 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:26.491639 32445 webserver.cc:460] Webserver started at http://127.28.191.3:37799/ using document root <none> and password file <none>
I20260806 02:40:26.492177 32445 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:26.492236 32445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:26.492481 32445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:26.494418 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "285a123b68874b2caa312b0672915611"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "a0cacf02575fd7377401f3922749aafb"
server_key_iv: "80f821701aac0218da904ce7b9605ac2"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.494988 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "285a123b68874b2caa312b0672915611"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "a0cacf02575fd7377401f3922749aafb"
server_key_iv: "80f821701aac0218da904ce7b9605ac2"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.498533 32445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 02:40:26.500754 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.502029 32445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:26.502130 32445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "285a123b68874b2caa312b0672915611"
format_stamp: "Formatted at 2026-08-06 02:40:26 on dist-test-slave-tfpf"
server_key: "a0cacf02575fd7377401f3922749aafb"
server_key_iv: "80f821701aac0218da904ce7b9605ac2"
server_key_version: "encryptionkey@0"
I20260806 02:40:26.502269 32445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:26.521348 32445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:26.524695 32445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:26.524919 32445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:26.526309 32445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:26.526357 32445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:26.526396 32445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:26.526455 32445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:26.538483 32445 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:37033
I20260806 02:40:26.538553 32645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:37033 every 8 connection(s)
I20260806 02:40:26.539446 32445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:26.541052 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.527168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:35423 (local address 127.28.191.62:33351)
0806 02:40:26.527298 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.527301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.528324 (+ 1023us) server_negotiation.cc:407] Connection header received
0806 02:40:26.529380 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.529394 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.529449 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.529504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.531117 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.532053 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.533548 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.533755 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.533874 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.537160 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.537168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.537171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.537197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.538883 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.539395 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.539400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.539401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.539483 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.539912 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.539918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.539922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.540214 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.540337 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.540796 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.540910 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:40:26.541885 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.527666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.3:35423)
0806 02:40:26.528173 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.528211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.529145 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.529943 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.529951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.530392 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.530953 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.530973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.532209 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.532216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.533435 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.533449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.533916 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.533920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.534108 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.534822 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.534837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.536944 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.539011 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.539017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.539032 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.539284 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.539629 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.539632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.539634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.539768 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.540340 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.540346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.540570 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.540723 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.540954 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":158,"threads_started":1}
I20260806 02:40:26.541961 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 32445
I20260806 02:40:26.542114 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:26.542495 29436 external_mini_cluster.cc:1428] Setting key 8ae0e5287d75fd1d5e2bd9b80d6380d1
I20260806 02:40:26.543473 32647 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33351
I20260806 02:40:26.543620 32647 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:26.544334 32647 heartbeater.cc:511] Master 127.28.191.62:33351 requested a full tablet report, sending...
I20260806 02:40:26.545409 31920 ts_manager.cc:194] Registered new tserver with Master: 285a123b68874b2caa312b0672915611 (127.28.191.3:37033)
I20260806 02:40:26.546166 31920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:35423
I20260806 02:40:26.553431 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.545808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:38599 (local address 127.28.191.62:33351)
0806 02:40:26.545934 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.545938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.546100 (+ 162us) server_negotiation.cc:407] Connection header received
0806 02:40:26.546211 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.546214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.546268 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.546336 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.547849 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.548329 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.549178 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.549413 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.549533 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.550587 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.550590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.550593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.550617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.551847 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.552277 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.552281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.552283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.552376 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.552655 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.552660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.552661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.552818 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.552922 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.553207 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.553302 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:40:26.553439 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.545741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.28.191.3:38599)
0806 02:40:26.546016 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.546032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.546137 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.546349 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.546352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.546544 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.546631 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.546636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.548466 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.548469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.549033 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.549041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.549529 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.549533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.549639 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.550116 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.550127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.550427 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.551963 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.551968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.551970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.552170 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.552480 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.552483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.552485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.552566 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.552913 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.552916 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.552998 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.553144 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.553272 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260806 02:40:26.556291 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:26.582254 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.574457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.28.191.62:33351)
0806 02:40:26.574582 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.574585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.574595 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:40:26.574625 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.574626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.574659 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.574759 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.575397 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.575858 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.576708 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.576953 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.577067 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.579344 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.579347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.579350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.579378 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.580563 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.580972 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.580977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.580979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.581098 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.581416 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.581419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.581420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.581593 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.581770 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.581989 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.582096 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:40:26.585006 31920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:26.588094 31920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:26.602607 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.596307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.0.0.1:52442)
0806 02:40:26.597243 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.597259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.597337 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.598265 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.598268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.598287 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:26.598421 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.598426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599882 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.599885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.601159 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.601187 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.602174 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.602177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.602272 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602294 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.602346 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.602416 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":195,"spinlock_wait_cycles":2432,"thread_start_us":85,"threads_started":1}
I20260806 02:40:26.602619 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.597071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.28.191.2:38117)
0806 02:40:26.597445 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.597461 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.597657 (+ 196us) server_negotiation.cc:407] Connection header received
0806 02:40:26.597792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.597798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.598011 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.598198 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:26.598534 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599709 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.601233 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.601823 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.601956 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602315 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.602368 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.602467 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260806 02:40:26.602499 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.596620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.0.0.1:57840)
0806 02:40:26.597545 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.597557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.597672 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.598207 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.598213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.598237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:26.598406 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.598412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599851 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.599857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.601050 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.601104 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.601957 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.601960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.602077 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602114 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.602217 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.602341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":48,"spinlock_wait_cycles":13568,"thread_start_us":77,"threads_started":1}
I20260806 02:40:26.603209 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.596280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.0.0.1:40118)
0806 02:40:26.596962 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.597013 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.597167 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.598324 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.598327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.598345 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:26.598412 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.598417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599499 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.599504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.600763 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.600773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.602149 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.602155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.602528 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602552 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.602615 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.603058 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":271,"spinlock_wait_cycles":2048,"thread_start_us":62,"threads_started":1}
I20260806 02:40:26.603278 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.597483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.28.191.1:40623)
0806 02:40:26.597856 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.597876 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.597896 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:40:26.597962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.597966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.598128 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.598585 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:26.598725 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599769 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.601327 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.602051 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.602566 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602709 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.602766 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.602848 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":97,"threads_started":1}
I20260806 02:40:26.603765 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.596429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.28.191.3:37033)
0806 02:40:26.597905 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.597914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.597931 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:40:26.598009 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.598012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.598183 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.598558 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:26.598675 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.599376 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.601148 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.601980 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.602259 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.602436 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.603470 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.603608 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":110,"threads_started":1}
I20260806 02:40:26.605820 32332 tablet_service.cc:1467] Processing CreateTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 (DEFAULT_TABLE table=test-table [id=93d665c979694e64b9ccc9d297a2afc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.605968 32128 tablet_service.cc:1467] Processing CreateTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 (DEFAULT_TABLE table=test-table [id=93d665c979694e64b9ccc9d297a2afc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.605969 32541 tablet_service.cc:1467] Processing CreateTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 (DEFAULT_TABLE table=test-table [id=93d665c979694e64b9ccc9d297a2afc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.606969 32541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d6d26ca5f44ec898f34c88d93d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.606966 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d6d26ca5f44ec898f34c88d93d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.607244 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d6d26ca5f44ec898f34c88d93d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.614429 32681 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Bootstrap starting.
I20260806 02:40:26.616281 32681 tablet_bootstrap.cc:654] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.616688 32679 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Bootstrap starting.
I20260806 02:40:26.617385 32683 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Bootstrap starting.
I20260806 02:40:26.617623 32681 log.cc:824] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:26.618824 32679 tablet_bootstrap.cc:654] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.619385 32683 tablet_bootstrap.cc:654] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.619674 32681 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: No bootstrap required, opened a new log
I20260806 02:40:26.619652 32679 log.cc:824] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:26.619915 32681 ts_tablet_manager.cc:1397] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:40:26.620280 32683 log.cc:824] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:26.621855 32679 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: No bootstrap required, opened a new log
I20260806 02:40:26.622071 32679 ts_tablet_manager.cc:1397] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:40:26.624083 32683 tablet_bootstrap.cc:492] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: No bootstrap required, opened a new log
I20260806 02:40:26.624256 32683 ts_tablet_manager.cc:1397] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:40:26.624230 32681 raft_consensus.cc:348] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.624537 32681 raft_consensus.cc:374] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.624593 32681 raft_consensus.cc:729] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2964edc64444e5e9599663671c844d6, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.625133 32681 consensus_queue.cc:260] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.625757 32223 heartbeater.cc:503] Master 127.28.191.62:33351 was elected leader, sending a full tablet report...
I20260806 02:40:26.625945 32679 raft_consensus.cc:348] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.626161 32681 ts_tablet_manager.cc:1428] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:40:26.626185 32679 raft_consensus.cc:374] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.626236 32679 raft_consensus.cc:729] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.626677 32679 consensus_queue.cc:260] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.627760 32430 heartbeater.cc:503] Master 127.28.191.62:33351 was elected leader, sending a full tablet report...
I20260806 02:40:26.628019 32683 raft_consensus.cc:348] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.628211 32679 ts_tablet_manager.cc:1428] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:40:26.628262 32683 raft_consensus.cc:374] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.628325 32683 raft_consensus.cc:729] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285a123b68874b2caa312b0672915611, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.628679 32683 consensus_queue.cc:260] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.629312 32647 heartbeater.cc:503] Master 127.28.191.62:33351 was elected leader, sending a full tablet report...
I20260806 02:40:26.629549 32683 ts_tablet_manager.cc:1428] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:40:26.689077 32693 raft_consensus.cc:482] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:26.689327 32693 raft_consensus.cc:504] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.690654 32693 leader_election.cc:288] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:26.694665 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.691042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:44109 (local address 127.28.191.1:40623)
0806 02:40:26.691176 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.691181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.691201 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:40:26.691255 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.691260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.691319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.691399 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:26.691732 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.692642 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.693735 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.694117 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.694252 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.694308 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.694481 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.694543 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:40:26.694679 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.690824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.28.191.3:44109)
0806 02:40:26.690937 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.690953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.691042 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.691435 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.691438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.691460 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:26.691604 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.691611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.692733 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.692739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.693602 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.693611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.694241 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.694244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.694346 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.694372 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.694464 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.694545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 02:40:26.695034 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.690913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.28.191.3:53553)
0806 02:40:26.691252 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.691264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.691356 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.691597 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.691599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.691614 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:26.692286 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.692297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.693223 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.693228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.693935 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.693943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.694610 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.694613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.694708 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.694733 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.694847 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.694906 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":160,"threads_started":1}
I20260806 02:40:26.695099 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.691033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:53553 (local address 127.28.191.2:38117)
0806 02:40:26.691157 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.691161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.691344 (+ 183us) server_negotiation.cc:407] Connection header received
0806 02:40:26.691440 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.691444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.691489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.691587 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:26.692445 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.693109 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.694071 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.694461 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.694613 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.694660 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.694887 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.694966 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:40:26.695271 32151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d6d26ca5f44ec898f34c88d93d1296" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6" is_pre_election: true
I20260806 02:40:26.695585 32151 raft_consensus.cc:2454] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:26.695664 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d6d26ca5f44ec898f34c88d93d1296" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e" is_pre_election: true
I20260806 02:40:26.695914 32360 raft_consensus.cc:2454] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:26.696046 32466 leader_election.cc:302] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285a123b68874b2caa312b0672915611, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:26.696332 32693 raft_consensus.cc:2781] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:26.696424 32693 raft_consensus.cc:482] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:26.696486 32693 raft_consensus.cc:3037] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:26.697603 32693 raft_consensus.cc:504] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.697966 32693 leader_election.cc:288] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Requested vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:26.698271 32151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d6d26ca5f44ec898f34c88d93d1296" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6"
I20260806 02:40:26.698370 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d6d26ca5f44ec898f34c88d93d1296" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e"
I20260806 02:40:26.698383 32151 raft_consensus.cc:3037] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:26.698503 32360 raft_consensus.cc:3037] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:26.699458 32151 raft_consensus.cc:2454] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:26.699627 32360 raft_consensus.cc:2454] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:26.699764 32466 leader_election.cc:302] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285a123b68874b2caa312b0672915611, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:26.699949 32693 raft_consensus.cc:2781] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:26.700179 32693 raft_consensus.cc:686] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 1 LEADER]: Becoming Leader. State: Replica: 285a123b68874b2caa312b0672915611, State: Running, Role: LEADER
I20260806 02:40:26.700511 32693 consensus_queue.cc:237] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } }
I20260806 02:40:26.703634 31920 catalog_manager.cc:5495] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285a123b68874b2caa312b0672915611 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:40:26 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:26.743314 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.734820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33351 (local address 127.0.0.1:60926)
0806 02:40:26.735296 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.735310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.735414 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.735671 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.735673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.736067 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:26.736162 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.736168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.736925 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.736927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.737567 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.737577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.737983 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.737985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.738046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.738610 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:26.738622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:26.740042 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:26.741905 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.741914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.741919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.742165 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.742421 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:26.742423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:26.742424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:26.742504 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:26.742843 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:26.742848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:26.742950 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.743043 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.743152 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":164,"threads_started":1}
I20260806 02:40:26.743544 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.734863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.28.191.62:33351)
0806 02:40:26.735032 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.735036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.735397 (+ 361us) server_negotiation.cc:407] Connection header received
0806 02:40:26.735504 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.735506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.735552 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.735624 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:26.736290 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.736827 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.737678 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.737882 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.737961 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.740202 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:26.740206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:26.740207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:26.740228 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:26.741802 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.742260 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.742263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.742265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.742337 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:26.742584 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:26.742587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:26.742588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:26.742751 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:26.742827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.743152 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.743267 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:40:26.747560 31920 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:60926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:40:26.748297 31920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:26.753262 32541 tablet_service.cc:1467] Processing CreateTablet for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=975fd9b88a164123b6b027f6fc78c7b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:26.753160 32128 tablet_service.cc:1467] Processing CreateTablet for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=975fd9b88a164123b6b027f6fc78c7b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:26.753329 32332 tablet_service.cc:1467] Processing CreateTablet for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=975fd9b88a164123b6b027f6fc78c7b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:26.753928 32541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.753928 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.754034 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66ce1dc9bc241ff9e3a57ddae4c7d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.756364 32679 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: Bootstrap starting.
I20260806 02:40:26.756752 32681 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: Bootstrap starting.
I20260806 02:40:26.756934 32683 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: Bootstrap starting.
I20260806 02:40:26.757457 32679 tablet_bootstrap.cc:654] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.757725 32681 tablet_bootstrap.cc:654] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.757984 32683 tablet_bootstrap.cc:654] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.759186 32679 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: No bootstrap required, opened a new log
I20260806 02:40:26.759277 32679 ts_tablet_manager.cc:1397] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:26.759305 32681 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: No bootstrap required, opened a new log
I20260806 02:40:26.759299 32683 tablet_bootstrap.cc:492] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: No bootstrap required, opened a new log
I20260806 02:40:26.759374 32681 ts_tablet_manager.cc:1397] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:26.759380 32683 ts_tablet_manager.cc:1397] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:26.759765 32683 raft_consensus.cc:348] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.759765 32681 raft_consensus.cc:348] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.759749 32679 raft_consensus.cc:348] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.759887 32679 raft_consensus.cc:374] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.759887 32681 raft_consensus.cc:374] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.759889 32683 raft_consensus.cc:374] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.759929 32683 raft_consensus.cc:729] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285a123b68874b2caa312b0672915611, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.759929 32681 raft_consensus.cc:729] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2964edc64444e5e9599663671c844d6, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.759929 32679 raft_consensus.cc:729] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.760056 32679 consensus_queue.cc:260] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.760058 32683 consensus_queue.cc:260] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.760074 32681 consensus_queue.cc:260] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.760308 32679 ts_tablet_manager.cc:1428] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:26.760308 32681 ts_tablet_manager.cc:1428] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:26.760332 32683 ts_tablet_manager.cc:1428] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:26.760547 32691 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.760561 32693 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.760667 32691 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.760696 32693 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.760861 32689 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.761026 32689 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.785772 32693 raft_consensus.cc:482] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:26.785902 32693 raft_consensus.cc:504] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.786362 32693 leader_election.cc:288] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:26.786803 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66ce1dc9bc241ff9e3a57ddae4c7d81" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e" is_pre_election: true
I20260806 02:40:26.786803 32151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66ce1dc9bc241ff9e3a57ddae4c7d81" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6" is_pre_election: true
I20260806 02:40:26.786953 32151 raft_consensus.cc:2454] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:26.786953 32360 raft_consensus.cc:2454] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:26.787307 32470 leader_election.cc:302] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, 285a123b68874b2caa312b0672915611; no voters:
I20260806 02:40:26.787537 32693 raft_consensus.cc:2781] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:26.787616 32693 raft_consensus.cc:482] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:26.787674 32693 raft_consensus.cc:3037] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:26.788655 32693 raft_consensus.cc:504] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.788987 32693 leader_election.cc:288] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Requested vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:26.789357 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66ce1dc9bc241ff9e3a57ddae4c7d81" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e"
I20260806 02:40:26.789354 32151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66ce1dc9bc241ff9e3a57ddae4c7d81" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6"
I20260806 02:40:26.789485 32360 raft_consensus.cc:3037] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:26.789485 32151 raft_consensus.cc:3037] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:40:26.790540 32648 tablet.cc:2369] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:26.790587 32360 raft_consensus.cc:2454] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:26.790593 32151 raft_consensus.cc:2454] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:26.790874 32470 leader_election.cc:302] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, 285a123b68874b2caa312b0672915611; no voters:
I20260806 02:40:26.791071 32693 raft_consensus.cc:2781] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:26.791164 32693 raft_consensus.cc:686] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [term 1 LEADER]: Becoming Leader. State: Replica: 285a123b68874b2caa312b0672915611, State: Running, Role: LEADER
I20260806 02:40:26.791321 32693 consensus_queue.cc:237] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.792066 32695 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: TxnStatusTablet state changed. Reason: New leader 285a123b68874b2caa312b0672915611. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.792187 32695 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:26.792498 32709 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:40:26.792788 32709 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:40:26.793257 31920 catalog_manager.cc:5495] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285a123b68874b2caa312b0672915611 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } health_report { overall_health: HEALTHY } } }
I20260806 02:40:26.806188 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.801854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.0.0.1:40128)
0806 02:40:26.802018 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.802036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.802136 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.802505 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.802507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.802514 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:26.802607 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.802611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.803621 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.803624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.804244 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.804253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.804844 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.804846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.804911 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.804956 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:26.805849 (+ 893us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:26.805866 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.805938 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.806052 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:40:26.806167 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.801933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.28.191.3:37033)
0806 02:40:26.802064 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.802069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.802087 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:40:26.802282 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.802286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.802354 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.802480 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:26.802725 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.803512 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.804368 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.804702 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.804843 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.805138 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:26.805670 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:26.805799 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.805973 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.806036 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:40:26.817454 32432 tablet.cc:2369] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:26.819000 32151 raft_consensus.cc:1261] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:26.819016 32360 raft_consensus.cc:1261] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:26.819706 32695 consensus_queue.cc:1035] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:26.819962 32693 consensus_queue.cc:1035] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:26.823871 32689 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: TxnStatusTablet state changed. Reason: New leader 285a123b68874b2caa312b0672915611. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.824018 32689 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.824230 32691 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: TxnStatusTablet state changed. Reason: New leader 285a123b68874b2caa312b0672915611. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.824349 32691 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.827282 32693 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.827392 32693 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:26.827895 32695 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.827996 32695 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 285a123b68874b2caa312b0672915611: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:26.828928 32691 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.829025 32691 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P 0c8d456875a54034afc6616c80c5127e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.829319 32717 mvcc.cc:205] Tried to move back new op lower bound from 7315390573841801216 to 7315390573738885120. Current Snapshot: MvccSnapshot[applied={T|T < 7315390573841801216}]
I20260806 02:40:26.829838 32689 tablet_replica.cc:439] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } } }
I20260806 02:40:26.829944 32689 tablet_replica.cc:442] T b66ce1dc9bc241ff9e3a57ddae4c7d81 P f2964edc64444e5e9599663671c844d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:26.832247 32716 mvcc.cc:205] Tried to move back new op lower bound from 7315390573841801216 to 7315390573738885120. Current Snapshot: MvccSnapshot[applied={T|T < 7315390573841801216}]
I20260806 02:40:26.838259 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.834968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.28.191.3:37033)
0806 02:40:26.835086 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.835091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.835127 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:40:26.835310 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.835314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.835360 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.835442 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:26.835735 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.836239 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.837183 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.837400 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.837515 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.837812 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:26.837884 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:26.837971 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.838078 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.838127 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:40:26.848850 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.845453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.28.191.3:56113)
0806 02:40:26.845672 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:26.845686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:26.845774 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:26.846008 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:26.846011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:26.846017 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:26.846136 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.846141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.846786 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.846790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.847719 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:26.847729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.848130 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.848133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:26.848221 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.848269 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:26.848574 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:26.848580 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:40:26.848644 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:40:26.848717 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 02:40:26.848861 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:26.845527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:56113 (local address 127.28.191.3:37033)
0806 02:40:26.845642 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:26.845645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:26.845742 (+ 97us) server_negotiation.cc:407] Connection header received
0806 02:40:26.845857 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:26.845859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:26.845895 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:26.845954 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:26.846231 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.846672 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.847844 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:26.848019 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:26.848088 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:26.848415 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:26.848468 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:26.848567 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:26.848674 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 02:40:26.848723 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 02:40:26.850983 32225 tablet.cc:2369] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:26.855101 32360 raft_consensus.cc:1261] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:26.855234 32151 raft_consensus.cc:1261] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:26.855472 32726 consensus_queue.cc:1035] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:26.855612 32695 consensus_queue.cc:1035] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:26.860716 32715 mvcc.cc:205] Tried to move back new op lower bound from 7315390573995134976 to 7315390573368365056. Current Snapshot: MvccSnapshot[applied={T|T < 7315390573995134976}]
I20260806 02:40:26.897157 31920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:26.897348 31920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:26.899757 31920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Sending DeleteTablet for 3 replicas of tablet e1d6d26ca5f44ec898f34c88d93d1296
I20260806 02:40:26.900427 32540 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:26 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:40118
I20260806 02:40:26.900507 32128 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:26 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260806 02:40:26.900854 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d6d26ca5f44ec898f34c88d93d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:26 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260806 02:40:26.900919 32734 tablet_replica.cc:330] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: stopping tablet replica
I20260806 02:40:26.901245 32735 tablet_replica.cc:330] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: stopping tablet replica
I20260806 02:40:26.901247 32734 raft_consensus.cc:2229] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:26.901511 32735 raft_consensus.cc:2229] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:26.901568 32734 raft_consensus.cc:2258] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:26.901686 32735 raft_consensus.cc:2258] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:26.902195 31920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:26.902611 31920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:26.902959 32735 ts_tablet_manager.cc:1905] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:26.902961 32734 ts_tablet_manager.cc:1905] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:26.906060 32735 ts_tablet_manager.cc:1918] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:26.906077 32734 ts_tablet_manager.cc:1918] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:26.906157 32735 log.cc:1196] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/e1d6d26ca5f44ec898f34c88d93d1296
I20260806 02:40:26.906179 32734 log.cc:1196] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/e1d6d26ca5f44ec898f34c88d93d1296
I20260806 02:40:26.906500 32735 ts_tablet_manager.cc:1939] T e1d6d26ca5f44ec898f34c88d93d1296 P f2964edc64444e5e9599663671c844d6: Deleting consensus metadata
I20260806 02:40:26.906577 32734 ts_tablet_manager.cc:1939] T e1d6d26ca5f44ec898f34c88d93d1296 P 285a123b68874b2caa312b0672915611: Deleting consensus metadata
I20260806 02:40:26.907668 31895 catalog_manager.cc:4841] TS 285a123b68874b2caa312b0672915611 (127.28.191.3:37033): tablet e1d6d26ca5f44ec898f34c88d93d1296 (table test-table [id=93d665c979694e64b9ccc9d297a2afc5]) successfully deleted
I20260806 02:40:26.907877 31893 catalog_manager.cc:4841] TS f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623): tablet e1d6d26ca5f44ec898f34c88d93d1296 (table test-table [id=93d665c979694e64b9ccc9d297a2afc5]) successfully deleted
I20260806 02:40:26.908253 32540 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 (DEFAULT_TABLE table=test-table [id=f185fddbefde41be9561548f7d46d004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.908314 32128 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 (DEFAULT_TABLE table=test-table [id=f185fddbefde41be9561548f7d46d004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.908514 32540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3cbd3f8e5e40c2be43b350653d3073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.908550 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3cbd3f8e5e40c2be43b350653d3073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.908577 32332 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 (DEFAULT_TABLE table=test-table [id=f185fddbefde41be9561548f7d46d004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:26.908814 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3cbd3f8e5e40c2be43b350653d3073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:26.910533 32683 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Bootstrap starting.
I20260806 02:40:26.910739 32681 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Bootstrap starting.
I20260806 02:40:26.911118 32679 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Bootstrap starting.
I20260806 02:40:26.911307 32683 tablet_bootstrap.cc:654] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.911604 32681 tablet_bootstrap.cc:654] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.912036 32679 tablet_bootstrap.cc:654] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:26.912611 32736 tablet_replica.cc:330] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: stopping tablet replica
I20260806 02:40:26.912873 32736 raft_consensus.cc:2229] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:26.912899 32681 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: No bootstrap required, opened a new log
I20260806 02:40:26.912966 32681 ts_tablet_manager.cc:1397] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:26.913043 32736 raft_consensus.cc:2258] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:26.913344 32681 raft_consensus.cc:348] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.913412 32679 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: No bootstrap required, opened a new log
I20260806 02:40:26.913443 32681 raft_consensus.cc:374] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.913471 32681 raft_consensus.cc:729] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2964edc64444e5e9599663671c844d6, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.913482 32679 ts_tablet_manager.cc:1397] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:26.913592 32681 consensus_queue.cc:260] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.913836 32681 ts_tablet_manager.cc:1428] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:26.913931 32679 raft_consensus.cc:348] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.914023 32679 raft_consensus.cc:374] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.914065 32679 raft_consensus.cc:729] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.914170 32679 consensus_queue.cc:260] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.914340 32736 ts_tablet_manager.cc:1905] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:26.914386 32679 ts_tablet_manager.cc:1428] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:26.916553 32683 tablet_bootstrap.cc:492] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: No bootstrap required, opened a new log
I20260806 02:40:26.916643 32683 ts_tablet_manager.cc:1397] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 02:40:26.917128 32683 raft_consensus.cc:348] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.917246 32683 raft_consensus.cc:374] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:26.917286 32683 raft_consensus.cc:729] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285a123b68874b2caa312b0672915611, State: Initialized, Role: FOLLOWER
I20260806 02:40:26.917433 32683 consensus_queue.cc:260] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:26.917507 32736 ts_tablet_manager.cc:1918] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:26.917698 32736 log.cc:1196] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/e1d6d26ca5f44ec898f34c88d93d1296
I20260806 02:40:26.917732 32683 ts_tablet_manager.cc:1428] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:26.918054 32736 ts_tablet_manager.cc:1939] T e1d6d26ca5f44ec898f34c88d93d1296 P 0c8d456875a54034afc6616c80c5127e: Deleting consensus metadata
I20260806 02:40:26.919183 31898 catalog_manager.cc:4841] TS 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117): tablet e1d6d26ca5f44ec898f34c88d93d1296 (table test-table [id=93d665c979694e64b9ccc9d297a2afc5]) successfully deleted
I20260806 02:40:27.072201 32691 raft_consensus.cc:482] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:27.072331 32691 raft_consensus.cc:504] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:27.073650 32691 leader_election.cc:288] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 285a123b68874b2caa312b0672915611 (127.28.191.3:37033)
I20260806 02:40:27.078296 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.074299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:41163 (local address 127.28.191.1:40623)
0806 02:40:27.074423 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:27.074427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:27.074445 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:40:27.074542 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:27.074545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:27.074594 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:27.074684 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:27.075002 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.075723 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077277 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.077655 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077796 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.077849 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:27.078082 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 02:40:27.078168 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:40:27.078420 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.074024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:37279 (local address 127.28.191.3:37033)
0806 02:40:27.074192 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:27.074200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:27.074274 (+ 74us) server_negotiation.cc:407] Connection header received
0806 02:40:27.074568 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:27.074571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:27.074613 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:27.074681 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:27.075018 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.075566 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077224 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.077840 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.078049 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:27.078223 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:40:27.078299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:40:27.078639 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.073907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.28.191.2:41163)
0806 02:40:27.074340 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:27.074352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:27.074434 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:27.074714 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:27.074717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:27.074733 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:27.074860 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.074866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.075840 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.075843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.077108 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.077120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.077796 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.077907 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.077938 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:40:27.078433 (+ 495us) client_negotiation.cc:240] Negotiation successful
0806 02:40:27.078497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260806 02:40:27.078682 32744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.073888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.28.191.2:37279)
0806 02:40:27.074190 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:27.074205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:27.074351 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:27.074817 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:27.074821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:27.074837 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:27.074925 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.074934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.075677 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.075682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.077108 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.077120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.077963 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.077969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.078095 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.078113 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:40:27.078180 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:40:27.078244 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":69,"thread_start_us":84,"threads_started":1}
I20260806 02:40:27.079174 32563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3cbd3f8e5e40c2be43b350653d3073" candidate_uuid: "0c8d456875a54034afc6616c80c5127e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285a123b68874b2caa312b0672915611" is_pre_election: true
I20260806 02:40:27.079337 32150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3cbd3f8e5e40c2be43b350653d3073" candidate_uuid: "0c8d456875a54034afc6616c80c5127e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6" is_pre_election: true
I20260806 02:40:27.079420 32563 raft_consensus.cc:2454] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8d456875a54034afc6616c80c5127e in term 0.
I20260806 02:40:27.079502 32150 raft_consensus.cc:2454] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8d456875a54034afc6616c80c5127e in term 0.
I20260806 02:40:27.079900 32265 leader_election.cc:302] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, 285a123b68874b2caa312b0672915611, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:27.080231 32691 raft_consensus.cc:2781] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:27.080309 32691 raft_consensus.cc:482] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:27.080354 32691 raft_consensus.cc:3037] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:27.081738 32691 raft_consensus.cc:504] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:27.082104 32691 leader_election.cc:288] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [CANDIDATE]: Term 1 election: Requested vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 285a123b68874b2caa312b0672915611 (127.28.191.3:37033)
I20260806 02:40:27.082495 32150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3cbd3f8e5e40c2be43b350653d3073" candidate_uuid: "0c8d456875a54034afc6616c80c5127e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6"
I20260806 02:40:27.082492 32563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3cbd3f8e5e40c2be43b350653d3073" candidate_uuid: "0c8d456875a54034afc6616c80c5127e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285a123b68874b2caa312b0672915611"
I20260806 02:40:27.082607 32150 raft_consensus.cc:3037] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:27.082612 32563 raft_consensus.cc:3037] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:27.083562 32563 raft_consensus.cc:2454] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8d456875a54034afc6616c80c5127e in term 1.
I20260806 02:40:27.083690 32150 raft_consensus.cc:2454] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8d456875a54034afc6616c80c5127e in term 1.
I20260806 02:40:27.083930 32265 leader_election.cc:302] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, 285a123b68874b2caa312b0672915611; no voters:
I20260806 02:40:27.084100 32691 raft_consensus.cc:2781] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:27.084331 32691 raft_consensus.cc:686] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 1 LEADER]: Becoming Leader. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Running, Role: LEADER
I20260806 02:40:27.084570 32691 consensus_queue.cc:237] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:27.086983 31918 catalog_manager.cc:5495] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8d456875a54034afc6616c80c5127e (127.28.191.2). New cstate: current_term: 1 leader_uuid: "0c8d456875a54034afc6616c80c5127e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:27.117379 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.113456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.28.191.2:38117)
0806 02:40:27.113601 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:27.113606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:27.113621 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:40:27.113736 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:27.113739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:27.113797 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:27.113894 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:27.114150 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.114963 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.115728 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.116050 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.116148 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.116378 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:27.116950 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:27.117091 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:27.117180 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:40:27.117237 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:27.127882 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.124536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:46549 (local address 127.28.191.3:37033)
0806 02:40:27.124686 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:27.124691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:27.124816 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:40:27.124980 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:27.124983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:27.125128 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:27.125211 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:27.125481 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.125965 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.126822 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.127057 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.127174 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.127455 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:27.127527 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:27.127612 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:27.127699 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:40:27.127745 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:40:27.128154 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.124388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.28.191.2:46549)
0806 02:40:27.124719 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:27.124734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:27.124867 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:27.125241 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:27.125244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:27.125251 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:27.125370 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.125375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.126089 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.126094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.126692 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.126700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.127174 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.127177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.127270 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.127318 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:27.127617 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:27.127626 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:40:27.127691 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:40:27.127766 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
I20260806 02:40:27.134094 32150 raft_consensus.cc:1261] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Refusing update from remote peer 0c8d456875a54034afc6616c80c5127e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:27.134306 32563 raft_consensus.cc:1261] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Refusing update from remote peer 0c8d456875a54034afc6616c80c5127e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:27.134544 32691 consensus_queue.cc:1035] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:27.134800 32748 consensus_queue.cc:1035] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [LEADER]: Connected to new peer: Peer: permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:27.151657 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.148595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:56341 (local address 127.28.191.2:38117)
0806 02:40:27.148696 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:27.148702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:27.148717 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:40:27.148883 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:27.148887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:27.148942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:27.149023 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:27.149298 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.149797 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.150623 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.150847 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.150960 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.151208 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:27.151328 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:27.151402 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:27.151470 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 02:40:27.151537 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:40:27.151692 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:27.148478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.28.191.3:56341)
0806 02:40:27.148662 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:27.148677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:27.148764 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:27.149080 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:27.149083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:27.149091 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:27.149183 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.149193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.149916 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.149920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.150499 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:27.150507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:27.150962 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:27.150966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:27.151055 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:27.151086 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:27.151399 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:27.151405 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:40:27.151460 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 02:40:27.151558 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
W20260806 02:40:27.152467 32716 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:40:28.709618 31917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:28.709766 31917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:28.712005 31917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Sending DeleteTablet for 3 replicas of tablet 4a3cbd3f8e5e40c2be43b350653d3073
I20260806 02:40:28.712518 32128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260806 02:40:28.712612 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260806 02:40:28.712602 32540 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3cbd3f8e5e40c2be43b350653d3073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:40118
I20260806 02:40:28.712937 327 tablet_replica.cc:330] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: stopping tablet replica
I20260806 02:40:28.712945 329 tablet_replica.cc:330] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: stopping tablet replica
I20260806 02:40:28.713114 327 raft_consensus.cc:2229] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:28.713596 327 raft_consensus.cc:2258] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.713132 329 raft_consensus.cc:2229] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:28.713820 329 raft_consensus.cc:2258] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.714193 327 ts_tablet_manager.cc:1905] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:28.714475 329 ts_tablet_manager.cc:1905] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:28.716590 331 tablet_replica.cc:330] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: stopping tablet replica
I20260806 02:40:28.716818 331 raft_consensus.cc:2229] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:28.717032 327 ts_tablet_manager.cc:1918] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:40:28.717154 327 log.cc:1196] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/4a3cbd3f8e5e40c2be43b350653d3073
I20260806 02:40:28.717196 331 raft_consensus.cc:2258] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.717450 327 ts_tablet_manager.cc:1939] T 4a3cbd3f8e5e40c2be43b350653d3073 P f2964edc64444e5e9599663671c844d6: Deleting consensus metadata
I20260806 02:40:28.718111 31893 catalog_manager.cc:4841] TS f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623): tablet 4a3cbd3f8e5e40c2be43b350653d3073 (table test-table [id=f185fddbefde41be9561548f7d46d004]) successfully deleted
I20260806 02:40:28.719448 329 ts_tablet_manager.cc:1918] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:40:28.719574 329 log.cc:1196] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/4a3cbd3f8e5e40c2be43b350653d3073
I20260806 02:40:28.719945 329 ts_tablet_manager.cc:1939] T 4a3cbd3f8e5e40c2be43b350653d3073 P 285a123b68874b2caa312b0672915611: Deleting consensus metadata
I20260806 02:40:28.720715 31895 catalog_manager.cc:4841] TS 285a123b68874b2caa312b0672915611 (127.28.191.3:37033): tablet 4a3cbd3f8e5e40c2be43b350653d3073 (table test-table [id=f185fddbefde41be9561548f7d46d004]) successfully deleted
I20260806 02:40:28.720794 331 ts_tablet_manager.cc:1905] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:28.723451 331 ts_tablet_manager.cc:1918] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:40:28.723551 331 log.cc:1196] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/4a3cbd3f8e5e40c2be43b350653d3073
I20260806 02:40:28.723783 331 ts_tablet_manager.cc:1939] T 4a3cbd3f8e5e40c2be43b350653d3073 P 0c8d456875a54034afc6616c80c5127e: Deleting consensus metadata
I20260806 02:40:28.725355 31898 catalog_manager.cc:4841] TS 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117): tablet 4a3cbd3f8e5e40c2be43b350653d3073 (table test-table [id=f185fddbefde41be9561548f7d46d004]) successfully deleted
Aug 06 02:40:28 dist-test-slave-tfpf krb5kdc[31855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:40:28 dist-test-slave-tfpf krb5kdc[31855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984028, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:28.747496 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:28.733980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.28.191.62:33351)
0806 02:40:28.734244 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:28.734262 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:28.734339 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:40:28.734490 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:28.734493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:28.734570 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:28.734679 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:28.735447 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:28.735993 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:28.736871 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:28.737203 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:28.737321 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:28.744295 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:28.744298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:28.744300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:28.744323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:28.745800 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:28.746227 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:28.746232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:28.746234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:28.746327 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:28.746595 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:28.746599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:28.746601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:28.746791 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:28.746962 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:28.747218 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:40:28.747337 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260806 02:40:28.750087 31917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:28.750568 31917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:28.755095 32128 tablet_service.cc:1467] Processing CreateTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f (DEFAULT_TABLE table=test-table [id=e1e210c402a444ec8f95a6f683eb4118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.755400 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8bc7e407e74cae8e6f604145eb1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.755424 32540 tablet_service.cc:1467] Processing CreateTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f (DEFAULT_TABLE table=test-table [id=e1e210c402a444ec8f95a6f683eb4118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.755708 32540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8bc7e407e74cae8e6f604145eb1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.757493 32332 tablet_service.cc:1467] Processing CreateTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f (DEFAULT_TABLE table=test-table [id=e1e210c402a444ec8f95a6f683eb4118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.757827 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8bc7e407e74cae8e6f604145eb1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.758013 343 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Bootstrap starting.
I20260806 02:40:28.758281 344 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Bootstrap starting.
I20260806 02:40:28.758945 343 tablet_bootstrap.cc:654] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.759228 344 tablet_bootstrap.cc:654] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.760223 343 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: No bootstrap required, opened a new log
I20260806 02:40:28.760321 343 ts_tablet_manager.cc:1397] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:28.760475 347 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Bootstrap starting.
I20260806 02:40:28.760835 343 raft_consensus.cc:348] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.760962 343 raft_consensus.cc:374] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.761003 343 raft_consensus.cc:729] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2964edc64444e5e9599663671c844d6, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.761171 343 consensus_queue.cc:260] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.761466 347 tablet_bootstrap.cc:654] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.761619 343 ts_tablet_manager.cc:1428] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:40:28.762630 344 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: No bootstrap required, opened a new log
I20260806 02:40:28.762673 347 tablet_bootstrap.cc:492] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: No bootstrap required, opened a new log
I20260806 02:40:28.762734 347 ts_tablet_manager.cc:1397] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:28.762734 344 ts_tablet_manager.cc:1397] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:40:28.763161 344 raft_consensus.cc:348] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.763264 344 raft_consensus.cc:374] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.763296 344 raft_consensus.cc:729] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285a123b68874b2caa312b0672915611, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.763306 347 raft_consensus.cc:348] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.763437 347 raft_consensus.cc:374] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.763419 344 consensus_queue.cc:260] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.763468 347 raft_consensus.cc:729] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.763605 347 consensus_queue.cc:260] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.763675 344 ts_tablet_manager.cc:1428] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:28.764357 347 ts_tablet_manager.cc:1428] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:28.791736 32761 raft_consensus.cc:482] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:28.791850 32761 raft_consensus.cc:504] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.792281 32761 leader_election.cc:288] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:28.792721 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8bc7e407e74cae8e6f604145eb1b1f" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e" is_pre_election: true
I20260806 02:40:28.792758 32150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8bc7e407e74cae8e6f604145eb1b1f" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6" is_pre_election: true
I20260806 02:40:28.792937 32150 raft_consensus.cc:2454] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:28.792932 32360 raft_consensus.cc:2454] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 0.
I20260806 02:40:28.793342 32470 leader_election.cc:302] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, 285a123b68874b2caa312b0672915611; no voters:
I20260806 02:40:28.793541 32761 raft_consensus.cc:2781] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:28.793606 32761 raft_consensus.cc:482] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:28.793661 32761 raft_consensus.cc:3037] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:28.794878 32761 raft_consensus.cc:504] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.795266 32761 leader_election.cc:288] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Requested vote from peers f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623), 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117)
I20260806 02:40:28.795565 32150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8bc7e407e74cae8e6f604145eb1b1f" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2964edc64444e5e9599663671c844d6"
I20260806 02:40:28.795661 32150 raft_consensus.cc:3037] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:28.795627 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8bc7e407e74cae8e6f604145eb1b1f" candidate_uuid: "285a123b68874b2caa312b0672915611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e"
I20260806 02:40:28.795763 32360 raft_consensus.cc:3037] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:28.796597 32150 raft_consensus.cc:2454] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:28.796700 32360 raft_consensus.cc:2454] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285a123b68874b2caa312b0672915611 in term 1.
I20260806 02:40:28.796903 32466 leader_election.cc:302] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285a123b68874b2caa312b0672915611, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:28.797139 32761 raft_consensus.cc:2781] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:28.797230 32761 raft_consensus.cc:686] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 1 LEADER]: Becoming Leader. State: Replica: 285a123b68874b2caa312b0672915611, State: Running, Role: LEADER
I20260806 02:40:28.797398 32761 consensus_queue.cc:237] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.799120 31917 catalog_manager.cc:5495] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285a123b68874b2caa312b0672915611 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "285a123b68874b2caa312b0672915611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } health_report { overall_health: HEALTHY } } }
I20260806 02:40:28.822518 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:28.816754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.28.191.3:37033)
0806 02:40:28.819292 (+ 2538us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:28.819298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:28.819310 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:28.819346 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:28.819348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:28.819428 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:28.819514 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:28.819891 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:28.820570 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:28.821582 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:28.821799 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:28.821880 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:28.822135 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:28.822200 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:28.822254 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:28.822373 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:40:28.822416 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474,"thread_start_us":117,"threads_started":1}
I20260806 02:40:28.828415 32150 raft_consensus.cc:1261] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:28.828415 32360 raft_consensus.cc:1261] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Refusing update from remote peer 285a123b68874b2caa312b0672915611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:28.828851 32693 consensus_queue.cc:1035] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:28.829042 32761 consensus_queue.cc:1035] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:28.857760 31917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:28.857889 31917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:28.859553 31917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Sending DeleteTablet for 3 replicas of tablet ac8bc7e407e74cae8e6f604145eb1b1f
I20260806 02:40:28.860080 32128 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260806 02:40:28.860105 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260806 02:40:28.860134 32541 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8bc7e407e74cae8e6f604145eb1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:40118
I20260806 02:40:28.860241 327 tablet_replica.cc:330] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: stopping tablet replica
I20260806 02:40:28.860255 331 tablet_replica.cc:330] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: stopping tablet replica
I20260806 02:40:28.860255 329 tablet_replica.cc:330] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: stopping tablet replica
I20260806 02:40:28.860338 327 raft_consensus.cc:2229] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:28.860344 329 raft_consensus.cc:2229] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:28.860342 331 raft_consensus.cc:2229] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:28.860422 327 raft_consensus.cc:2258] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.860423 331 raft_consensus.cc:2258] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.860514 329 raft_consensus.cc:2258] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:28.860918 31917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:40:28.861157 329 ts_tablet_manager.cc:1905] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:28.861253 331 ts_tablet_manager.cc:1905] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:40:28.861287 31917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:28.863791 331 ts_tablet_manager.cc:1918] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:28.863806 329 ts_tablet_manager.cc:1918] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:28.863842 331 log.cc:1196] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/ac8bc7e407e74cae8e6f604145eb1b1f
I20260806 02:40:28.863873 329 log.cc:1196] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/ac8bc7e407e74cae8e6f604145eb1b1f
I20260806 02:40:28.864082 331 ts_tablet_manager.cc:1939] T ac8bc7e407e74cae8e6f604145eb1b1f P 0c8d456875a54034afc6616c80c5127e: Deleting consensus metadata
I20260806 02:40:28.864146 329 ts_tablet_manager.cc:1939] T ac8bc7e407e74cae8e6f604145eb1b1f P 285a123b68874b2caa312b0672915611: Deleting consensus metadata
I20260806 02:40:28.864859 31898 catalog_manager.cc:4841] TS 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117): tablet ac8bc7e407e74cae8e6f604145eb1b1f (table test-table [id=e1e210c402a444ec8f95a6f683eb4118]) successfully deleted
I20260806 02:40:28.865154 31895 catalog_manager.cc:4841] TS 285a123b68874b2caa312b0672915611 (127.28.191.3:37033): tablet ac8bc7e407e74cae8e6f604145eb1b1f (table test-table [id=e1e210c402a444ec8f95a6f683eb4118]) successfully deleted
I20260806 02:40:28.865391 327 ts_tablet_manager.cc:1905] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:28.865957 32332 tablet_service.cc:1467] Processing CreateTablet for tablet e9348b19f6e148a1a333f92f53acfac9 (DEFAULT_TABLE table=test-table [id=274eec3976c945ca9e0bf10ac9289285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.865984 32128 tablet_service.cc:1467] Processing CreateTablet for tablet e9348b19f6e148a1a333f92f53acfac9 (DEFAULT_TABLE table=test-table [id=274eec3976c945ca9e0bf10ac9289285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.866163 32332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9348b19f6e148a1a333f92f53acfac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.866179 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9348b19f6e148a1a333f92f53acfac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.867887 327 ts_tablet_manager.cc:1918] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:28.867936 343 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: Bootstrap starting.
I20260806 02:40:28.867956 327 log.cc:1196] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/ac8bc7e407e74cae8e6f604145eb1b1f
I20260806 02:40:28.868239 347 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: Bootstrap starting.
I20260806 02:40:28.868310 327 ts_tablet_manager.cc:1939] T ac8bc7e407e74cae8e6f604145eb1b1f P f2964edc64444e5e9599663671c844d6: Deleting consensus metadata
I20260806 02:40:28.868918 343 tablet_bootstrap.cc:654] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.868866 32541 tablet_service.cc:1467] Processing CreateTablet for tablet e9348b19f6e148a1a333f92f53acfac9 (DEFAULT_TABLE table=test-table [id=274eec3976c945ca9e0bf10ac9289285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:28.869050 347 tablet_bootstrap.cc:654] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.869182 32541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9348b19f6e148a1a333f92f53acfac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:28.869789 31893 catalog_manager.cc:4841] TS f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623): tablet ac8bc7e407e74cae8e6f604145eb1b1f (table test-table [id=e1e210c402a444ec8f95a6f683eb4118]) successfully deleted
I20260806 02:40:28.870031 347 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: No bootstrap required, opened a new log
I20260806 02:40:28.870121 347 ts_tablet_manager.cc:1397] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:28.870334 343 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: No bootstrap required, opened a new log
I20260806 02:40:28.870436 343 ts_tablet_manager.cc:1397] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:40:28.870550 347 raft_consensus.cc:348] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.870659 347 raft_consensus.cc:374] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.870687 347 raft_consensus.cc:729] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8d456875a54034afc6616c80c5127e, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.870802 347 consensus_queue.cc:260] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.870854 343 raft_consensus.cc:348] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.870994 343 raft_consensus.cc:374] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.871033 343 raft_consensus.cc:729] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2964edc64444e5e9599663671c844d6, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.871061 347 ts_tablet_manager.cc:1428] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:28.871133 343 consensus_queue.cc:260] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.871369 344 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Bootstrap starting.
I20260806 02:40:28.871398 343 ts_tablet_manager.cc:1428] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:28.872174 344 tablet_bootstrap.cc:654] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:28.873346 344 tablet_bootstrap.cc:492] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: No bootstrap required, opened a new log
I20260806 02:40:28.873430 344 ts_tablet_manager.cc:1397] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:28.873811 344 raft_consensus.cc:348] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.873888 344 raft_consensus.cc:374] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:28.873907 344 raft_consensus.cc:729] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285a123b68874b2caa312b0672915611, State: Initialized, Role: FOLLOWER
I20260806 02:40:28.873989 344 consensus_queue.cc:260] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:28.874264 344 ts_tablet_manager.cc:1428] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:40:29.256279 348 raft_consensus.cc:482] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:29.256418 348 raft_consensus.cc:504] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:29.257786 348 leader_election.cc:288] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117), 285a123b68874b2caa312b0672915611 (127.28.191.3:37033)
I20260806 02:40:29.262017 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.258293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:48317 (local address 127.28.191.3:37033)
0806 02:40:29.258421 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:29.258426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:29.258667 (+ 241us) server_negotiation.cc:407] Connection header received
0806 02:40:29.258833 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:29.258836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:29.258882 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:29.258963 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:29.259291 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.259726 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.260865 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.261298 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.261387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.261487 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:29.261815 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:40:29.261897 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:40:29.262037 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.258116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:46037 (local address 127.28.191.2:38117)
0806 02:40:29.258394 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:29.258401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:29.258421 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:40:29.258618 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:29.258624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:29.258696 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:29.258789 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:29.259128 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.259713 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.260838 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.261323 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.261500 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.261557 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:29.261755 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 02:40:29.261883 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260806 02:40:29.262063 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.258114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.28.191.1:48317)
0806 02:40:29.258546 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:29.258565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:29.258695 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:29.259006 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:29.259009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:29.259026 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:29.259184 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.259191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.259867 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.259870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.260711 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.260720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.261428 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.261433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.261581 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.261665 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:40:29.261779 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:40:29.261851 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":51,"thread_start_us":110,"threads_started":1}
I20260806 02:40:29.262046 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.257976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.28.191.1:46037)
0806 02:40:29.258329 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:29.258347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:29.258464 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:29.258827 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:29.258830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:29.258851 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:29.259004 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.259011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.259855 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.259858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.260703 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.260712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.261497 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.261503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.261620 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.261640 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:40:29.261702 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:40:29.261790 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":52,"thread_start_us":90,"threads_started":1}
I20260806 02:40:29.262513 32563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9348b19f6e148a1a333f92f53acfac9" candidate_uuid: "f2964edc64444e5e9599663671c844d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285a123b68874b2caa312b0672915611" is_pre_election: true
I20260806 02:40:29.262733 32563 raft_consensus.cc:2454] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2964edc64444e5e9599663671c844d6 in term 0.
I20260806 02:40:29.263001 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9348b19f6e148a1a333f92f53acfac9" candidate_uuid: "f2964edc64444e5e9599663671c844d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e" is_pre_election: true
I20260806 02:40:29.263136 32359 raft_consensus.cc:2454] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2964edc64444e5e9599663671c844d6 in term 0.
I20260806 02:40:29.263266 32059 leader_election.cc:302] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285a123b68874b2caa312b0672915611, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:29.263536 348 raft_consensus.cc:2781] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:29.263589 348 raft_consensus.cc:482] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:29.263614 348 raft_consensus.cc:3037] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:29.264884 348 raft_consensus.cc:504] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:29.265318 348 leader_election.cc:288] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8d456875a54034afc6616c80c5127e (127.28.191.2:38117), 285a123b68874b2caa312b0672915611 (127.28.191.3:37033)
I20260806 02:40:29.265609 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9348b19f6e148a1a333f92f53acfac9" candidate_uuid: "f2964edc64444e5e9599663671c844d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8d456875a54034afc6616c80c5127e"
I20260806 02:40:29.265667 32563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9348b19f6e148a1a333f92f53acfac9" candidate_uuid: "f2964edc64444e5e9599663671c844d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285a123b68874b2caa312b0672915611"
I20260806 02:40:29.265749 32359 raft_consensus.cc:3037] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:29.265786 32563 raft_consensus.cc:3037] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:29.266671 32359 raft_consensus.cc:2454] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2964edc64444e5e9599663671c844d6 in term 1.
I20260806 02:40:29.266736 32563 raft_consensus.cc:2454] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2964edc64444e5e9599663671c844d6 in term 1.
I20260806 02:40:29.267009 32060 leader_election.cc:302] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8d456875a54034afc6616c80c5127e, f2964edc64444e5e9599663671c844d6; no voters:
I20260806 02:40:29.267177 348 raft_consensus.cc:2781] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:29.267421 348 raft_consensus.cc:686] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [term 1 LEADER]: Becoming Leader. State: Replica: f2964edc64444e5e9599663671c844d6, State: Running, Role: LEADER
I20260806 02:40:29.267678 348 consensus_queue.cc:237] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } }
I20260806 02:40:29.269806 31917 catalog_manager.cc:5495] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2964edc64444e5e9599663671c844d6 (127.28.191.1). New cstate: current_term: 1 leader_uuid: "f2964edc64444e5e9599663671c844d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2964edc64444e5e9599663671c844d6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:29.335321 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.331291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.28.191.1:40623)
0806 02:40:29.331634 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:29.331642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:29.331676 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:40:29.331876 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:29.331879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:29.331945 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:29.332080 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:29.332327 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.333175 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.334033 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.334251 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.334328 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.334574 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:29.335005 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:29.335094 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:29.335162 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 02:40:29.335205 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":139,"threads_started":1}
I20260806 02:40:29.345726 372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.341991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37033 (local address 127.28.191.1:41749)
0806 02:40:29.342355 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:29.342374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:29.342494 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:29.342790 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:29.342793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:29.342801 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:29.342926 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.342931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.343707 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.343713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.344410 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.344437 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.344993 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.344997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.345103 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.345143 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:29.345448 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:29.345456 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:40:29.345519 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:40:29.345595 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260806 02:40:29.345705 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.342128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:41749 (local address 127.28.191.3:37033)
0806 02:40:29.342268 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:29.342273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:29.342447 (+ 174us) server_negotiation.cc:407] Connection header received
0806 02:40:29.342602 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:29.342605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:29.342657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:29.342768 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:29.343037 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.343598 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.344567 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.344848 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.344977 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.345289 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:29.345356 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:29.345439 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:29.345528 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:40:29.345578 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:40:29.351398 32563 raft_consensus.cc:1261] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Refusing update from remote peer f2964edc64444e5e9599663671c844d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:29.351439 32359 raft_consensus.cc:1261] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Refusing update from remote peer f2964edc64444e5e9599663671c844d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:29.351950 363 consensus_queue.cc:1035] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285a123b68874b2caa312b0672915611" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:29.352378 348 consensus_queue.cc:1035] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8d456875a54034afc6616c80c5127e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:29.370265 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.366737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:35495 (local address 127.28.191.1:40623)
0806 02:40:29.366845 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:29.366850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:29.367074 (+ 224us) server_negotiation.cc:407] Connection header received
0806 02:40:29.367228 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:29.367231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:29.367284 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:29.367375 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:29.367662 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.368191 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.369130 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.369371 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.369500 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.369805 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:29.369883 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:29.369981 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:29.370080 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 02:40:29.370132 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 02:40:29.370306 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:29.366698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.28.191.3:35495)
0806 02:40:29.366967 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:29.366985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:29.367090 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:29.367386 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:29.367390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:29.367398 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:29.367541 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.367548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.368318 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.368323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.368981 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:29.368996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:29.369506 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:29.369510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:29.369629 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:29.369657 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:29.369991 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:29.370001 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:40:29.370071 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:40:29.370158 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
W20260806 02:40:29.371078 32763 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:40:30.928210 31918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:30.928354 31918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:30.930631 31918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afd0f15865014998a864d84c58424f64: Sending DeleteTablet for 3 replicas of tablet e9348b19f6e148a1a333f92f53acfac9
I20260806 02:40:30.931150 32128 tablet_service.cc:1514] Processing DeleteTablet for tablet e9348b19f6e148a1a333f92f53acfac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:30 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260806 02:40:30.931247 32332 tablet_service.cc:1514] Processing DeleteTablet for tablet e9348b19f6e148a1a333f92f53acfac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:30 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260806 02:40:30.931464 422 tablet_replica.cc:330] T e9348b19f6e148a1a333f92f53acfac9 P f2964edc64444e5e9599663671c844d6: stopping tablet replica
I20260806 02:40:30.931579 423 tablet_replica.cc:330] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: stopping tablet replica
I20260806 02:40:30.931691 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 32033
I20260806 02:40:30.931787 423 raft_consensus.cc:2229] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:30.937757 423 raft_consensus.cc:2258] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:30.938318 32541 tablet_service.cc:1514] Processing DeleteTablet for tablet e9348b19f6e148a1a333f92f53acfac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:30 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:40118
I20260806 02:40:30.938416 423 ts_tablet_manager.cc:1905] T e9348b19f6e148a1a333f92f53acfac9 P 0c8d456875a54034afc6616c80c5127e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:30.938647 424 tablet_replica.cc:330] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: stopping tablet replica
I20260806 02:40:30.938809 424 raft_consensus.cc:2229] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:30.938967 424 raft_consensus.cc:2258] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:30.939644 424 ts_tablet_manager.cc:1905] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:40:30.945638 31893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:40623 (error 108)
I20260806 02:40:30.945827 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 32235
I20260806 02:40:30.952054 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:30.950750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.0.0.1:52466)
0806 02:40:30.951770 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:30.951890 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":89,"threads_started":1}
W20260806 02:40:30.952317 31893 catalog_manager.cc:4584] TS f2964edc64444e5e9599663671c844d6 (127.28.191.1:40623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9348b19f6e148a1a333f92f53acfac9: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40623: connect: Connection refused (error 111)
I20260806 02:40:31.005951 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:31.005528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.0.0.1:52474)
0806 02:40:31.005747 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:31.005830 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:40:31.006981 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:31.006679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40623 (local address 127.0.0.1:52484)
0806 02:40:31.006822 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:31.006887 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 02:40:31.008790 424 ts_tablet_manager.cc:1918] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:40:31.008894 424 log.cc:1196] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/e9348b19f6e148a1a333f92f53acfac9
I20260806 02:40:31.009222 424 ts_tablet_manager.cc:1939] T e9348b19f6e148a1a333f92f53acfac9 P 285a123b68874b2caa312b0672915611: Deleting consensus metadata
I20260806 02:40:31.011926 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 32445
I20260806 02:40:31.014859 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:31.014603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.0.0.1:57860)
0806 02:40:31.014735 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:31.014789 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:40:31.047044 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:31.046680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.0.0.1:57862)
0806 02:40:31.046856 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:31.046946 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:40:31.047752 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:31.047550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:38117 (local address 127.0.0.1:57864)
0806 02:40:31.047653 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:31.047696 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:40:31.057229 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 31870
2026-08-06T02:40:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6749 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:32 dist-test-slave-tfpf krb5kdc[430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:32 dist-test-slave-tfpf krb5kdc[430](info): set up 2 sockets
Aug 06 02:40:32 dist-test-slave-tfpf krb5kdc[430](info): commencing operation
krb5kdc: starting...
W20260806 02:40:33.195168 29436 mini_kdc.cc:121] Time spent starting KDC: real 2.106s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:33.320657 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35707
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:33.434845 447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:33.435174 447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:33.435256 447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:33.439266 447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:33.439368 447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:33.439409 447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:33.439474 447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:33.439505 447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:33.444604 447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:33.445645 447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:33.446633 447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:33.455088 454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:33.455101 455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.455276 447 server_base.cc:1034] running on GCE node
W20260806 02:40:33.455240 457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.455947 447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:33.457003 447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:33.458150 447 hybrid_clock.cc:648] HybridClock initialized: now 1785984033458139 us; error 35 us; skew 500 ppm
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:33.460958 447 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:33.461845 447 webserver.cc:460] Webserver started at http://127.28.191.62:33353/ using document root <none> and password file <none>
I20260806 02:40:33.462400 447 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:33.462505 447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:33.462751 447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:33.464635 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "4673d789c93b4d1d91f5ae3d14ce04c7"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "606dfb9c352ad93904385b3ef67413a0"
server_key_iv: "324d4d4726a4b530e9e55fbe63d38217"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.465286 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "4673d789c93b4d1d91f5ae3d14ce04c7"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "606dfb9c352ad93904385b3ef67413a0"
server_key_iv: "324d4d4726a4b530e9e55fbe63d38217"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.468827 447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:40:33.471118 467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.472222 447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:33.472374 447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "4673d789c93b4d1d91f5ae3d14ce04c7"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "606dfb9c352ad93904385b3ef67413a0"
server_key_iv: "324d4d4726a4b530e9e55fbe63d38217"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.472492 447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:33.492201 447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:33.494975 447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:33.495194 447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:33.502290 447 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35707
I20260806 02:40:33.502348 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35707 every 8 connection(s)
I20260806 02:40:33.503278 447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:33.506150 545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:33.507460 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 447
I20260806 02:40:33.507571 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:33.507879 29436 external_mini_cluster.cc:1428] Setting key 4a47d1b61f00f3132e127114dc5e398a
I20260806 02:40:33.512349 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7: Bootstrap starting.
I20260806 02:40:33.514993 545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:33.515802 545 log.cc:824] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:33.517904 545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7: No bootstrap required, opened a new log
I20260806 02:40:33.521559 545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } }
I20260806 02:40:33.521718 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:33.509233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.28.191.62:35707)
0806 02:40:33.509603 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:33.509618 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:33.509650 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:40:33.510265 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:33.510280 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:33.510562 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:33.510809 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:33.511760 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.512795 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.514022 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.514462 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.514575 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:33.517306 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:33.517312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:33.517324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:33.517349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:33.519088 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:33.519550 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:33.519555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:33.519559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:33.519663 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:33.519967 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:33.519970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:33.519972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:33.520314 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:33.520525 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:33.520748 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 02:40:33.520999 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
I20260806 02:40:33.521824 545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:33.521919 545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4673d789c93b4d1d91f5ae3d14ce04c7, State: Initialized, Role: FOLLOWER
I20260806 02:40:33.522450 545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } }
I20260806 02:40:33.522631 545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:33.522703 545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:33.522807 545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:33.523736 545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } }
I20260806 02:40:33.524119 545 leader_election.cc:302] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4673d789c93b4d1d91f5ae3d14ce04c7; no voters:
I20260806 02:40:33.524430 545 leader_election.cc:288] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:33.524552 551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:33.524786 551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [term 1 LEADER]: Becoming Leader. State: Replica: 4673d789c93b4d1d91f5ae3d14ce04c7, State: Running, Role: LEADER
I20260806 02:40:33.525147 551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } }
I20260806 02:40:33.525733 545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:33.526942 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4673d789c93b4d1d91f5ae3d14ce04c7. Latest consensus state: current_term: 1 leader_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } } }
I20260806 02:40:33.526949 552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4673d789c93b4d1d91f5ae3d14ce04c7" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35707 } } }
I20260806 02:40:33.527082 552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:33.527082 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:33.528221 568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:33.531059 568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:33.536192 568 catalog_manager.cc:1347] Generated new cluster ID: 4bd7c35020834563b81c9d1b7502511b
I20260806 02:40:33.536264 568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:33.543010 568 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:33.543802 568 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:33.548277 568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4673d789c93b4d1d91f5ae3d14ce04c7: Generated new TSK 0
I20260806 02:40:33.548967 568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:33.586072 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:33.695720 581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:33.695977 581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:33.696045 581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:33.699944 581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:33.700037 581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:33.700122 581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:33.704825 581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:33.705861 581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:33.706763 581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:33.716177 587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:33.716177 588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.716276 581 server_base.cc:1034] running on GCE node
W20260806 02:40:33.716230 590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.716909 581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:33.717543 581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:33.718705 581 hybrid_clock.cc:648] HybridClock initialized: now 1785984033718676 us; error 40 us; skew 500 ppm
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:33.721474 581 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:33.722365 581 webserver.cc:460] Webserver started at http://127.28.191.1:43075/ using document root <none> and password file <none>
I20260806 02:40:33.722913 581 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:33.723009 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:33.723268 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:33.725281 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "947baa5a6d5a4156b13c40ede4bbbe45"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a182b0b62f995387e6545237c37e12b5"
server_key_iv: "795394e7ad04c12cc6c685786b879c16"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.725963 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "947baa5a6d5a4156b13c40ede4bbbe45"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a182b0b62f995387e6545237c37e12b5"
server_key_iv: "795394e7ad04c12cc6c685786b879c16"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.729666 581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:40:33.731920 604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.733166 581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:33.733259 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "947baa5a6d5a4156b13c40ede4bbbe45"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a182b0b62f995387e6545237c37e12b5"
server_key_iv: "795394e7ad04c12cc6c685786b879c16"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.733340 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:33.748947 581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:33.752068 581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:33.752229 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:33.752841 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:33.753734 581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:33.753780 581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.753851 581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:33.753882 581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.762315 581 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42281
I20260806 02:40:33.762399 772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42281 every 8 connection(s)
I20260806 02:40:33.763193 581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:33.772691 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 581
I20260806 02:40:33.772781 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:33.773164 29436 external_mini_cluster.cc:1428] Setting key 8ba89a9c05b379adcc7e781de954389f
I20260806 02:40:33.777305 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:33.764905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:43753 (local address 127.28.191.62:35707)
0806 02:40:33.765027 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:33.765031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:33.765868 (+ 837us) server_negotiation.cc:407] Connection header received
0806 02:40:33.767169 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:33.767173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:33.767231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:33.767346 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:33.768427 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.768953 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.770098 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.770329 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.770454 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:33.773005 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:33.773008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:33.773010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:33.773047 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:33.774812 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:33.775445 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:33.775450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:33.775452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:33.775575 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:33.775918 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:33.775923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:33.775925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:33.776119 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:33.776274 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:33.776967 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 02:40:33.777135 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:40:33.778108 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:33.765260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35707 (local address 127.28.191.1:43753)
0806 02:40:33.765722 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:33.765762 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:33.766951 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:33.767511 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:33.767517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:33.767822 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:33.768271 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:33.768284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.769123 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.769127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:33.769947 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:33.769957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:33.770489 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:33.770493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:33.770666 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:33.771351 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:33.771367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:33.772804 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:33.774938 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:33.774943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:33.774956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:33.775335 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:33.775661 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:33.775664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:33.775666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:33.775812 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:33.776308 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:33.776317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:33.776673 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 02:40:33.776920 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 02:40:33.777262 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260806 02:40:33.779326 773 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35707
I20260806 02:40:33.779574 773 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:33.780062 773 heartbeater.cc:511] Master 127.28.191.62:35707 requested a full tablet report, sending...
I20260806 02:40:33.781755 491 ts_manager.cc:194] Registered new tserver with Master: 947baa5a6d5a4156b13c40ede4bbbe45 (127.28.191.1:42281)
I20260806 02:40:33.783437 491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:43753
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:33.802207 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:33.913748 778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:33.913977 778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:33.914059 778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:33.917986 778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:33.918051 778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:33.918169 778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:33.923210 778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:33.924140 778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:33.925156 778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:33.934355 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.934379 778 server_base.cc:1034] running on GCE node
W20260806 02:40:33.934332 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:33.934324 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:33.934916 778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:33.935540 778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:33.936761 778 hybrid_clock.cc:648] HybridClock initialized: now 1785984033936741 us; error 29 us; skew 500 ppm
Aug 06 02:40:33 dist-test-slave-tfpf krb5kdc[430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:33.939559 778 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:33.940316 778 webserver.cc:460] Webserver started at http://127.28.191.2:42167/ using document root <none> and password file <none>
I20260806 02:40:33.940795 778 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:33.940850 778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:33.941010 778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:33.942927 778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "de7e049774b6484ea25340a8023f3bfd"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a50512d67c023dca1931b5ce0641c1fe"
server_key_iv: "155b394420737200aa95c9f56d8e954d"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.943428 778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "de7e049774b6484ea25340a8023f3bfd"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a50512d67c023dca1931b5ce0641c1fe"
server_key_iv: "155b394420737200aa95c9f56d8e954d"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.946849 778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:33.948987 794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.950107 778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:33.950240 778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "de7e049774b6484ea25340a8023f3bfd"
format_stamp: "Formatted at 2026-08-06 02:40:33 on dist-test-slave-tfpf"
server_key: "a50512d67c023dca1931b5ce0641c1fe"
server_key_iv: "155b394420737200aa95c9f56d8e954d"
server_key_version: "encryptionkey@0"
I20260806 02:40:33.950335 778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:33.989516 778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:33.992381 778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:33.992602 778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:33.993234 778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:33.994071 778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:33.994118 778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:33.994204 778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:33.994238 778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:34.003405 778 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:36663
I20260806 02:40:34.003429 969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:36663 every 8 connection(s)
I20260806 02:40:34.004318 778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:34.009271 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 778
I20260806 02:40:34.009421 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:34.009704 29436 external_mini_cluster.cc:1428] Setting key 8f2f38fc562817e0331b9fe42c6bebd4
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:34.019229 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:34.006282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:43147 (local address 127.28.191.62:35707)
0806 02:40:34.006428 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:34.006433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:34.007278 (+ 845us) server_negotiation.cc:407] Connection header received
0806 02:40:34.008508 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:34.008511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:34.008555 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:34.008630 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:34.010012 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.010665 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.011936 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.012325 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.012453 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:34.015197 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:34.015206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:34.015208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:34.015231 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:34.017126 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:34.017617 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:34.017624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:34.017628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:34.017728 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:34.018072 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:34.018078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:34.018081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:34.018339 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:34.018445 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:34.018883 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 02:40:34.019030 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:40:34.019867 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:34.006653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35707 (local address 127.28.191.2:43147)
0806 02:40:34.007115 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:34.007156 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:34.008317 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:34.008799 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:34.008805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:34.009253 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:34.009858 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:34.009873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.010803 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.010806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:34.011774 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:34.011784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.012473 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.012476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:34.012619 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:34.013304 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:34.013329 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:34.014990 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:34.017242 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:34.017246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:34.017257 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:34.017504 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:34.017831 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:34.017833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:34.017835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:34.017971 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:34.018441 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:34.018445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:34.018651 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 02:40:34.018797 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:40:34.019020 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:40:34.021193 970 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35707
I20260806 02:40:34.021454 970 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:34.021992 970 heartbeater.cc:511] Master 127.28.191.62:35707 requested a full tablet report, sending...
I20260806 02:40:34.023064 491 ts_manager.cc:194] Registered new tserver with Master: de7e049774b6484ea25340a8023f3bfd (127.28.191.2:36663)
I20260806 02:40:34.023643 491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:43147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:34.038378 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:34.147955 977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:34.148245 977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:34.148310 977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:34.152280 977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:34.152369 977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:34.152455 977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:34.157591 977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:34.158565 977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:34.159533 977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:34.168627 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:34.168681 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:34.168742 977 server_base.cc:1034] running on GCE node
W20260806 02:40:34.168967 990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:34.169360 977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:34.170001 977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:34.171186 977 hybrid_clock.cc:648] HybridClock initialized: now 1785984034171163 us; error 38 us; skew 500 ppm
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:34.173858 977 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:34.174707 977 webserver.cc:460] Webserver started at http://127.28.191.3:38497/ using document root <none> and password file <none>
I20260806 02:40:34.175233 977 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:34.175326 977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:34.175575 977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:34.177646 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "8cd06ff9259a4ebebd516044ff8a2aca"
format_stamp: "Formatted at 2026-08-06 02:40:34 on dist-test-slave-tfpf"
server_key: "3f28462303a41addb47f71c363596fd4"
server_key_iv: "b2d76838276749e808dc836c2c685304"
server_key_version: "encryptionkey@0"
I20260806 02:40:34.178208 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "8cd06ff9259a4ebebd516044ff8a2aca"
format_stamp: "Formatted at 2026-08-06 02:40:34 on dist-test-slave-tfpf"
server_key: "3f28462303a41addb47f71c363596fd4"
server_key_iv: "b2d76838276749e808dc836c2c685304"
server_key_version: "encryptionkey@0"
I20260806 02:40:34.181828 977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:34.184326 1002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:34.185628 977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:34.185761 977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "8cd06ff9259a4ebebd516044ff8a2aca"
format_stamp: "Formatted at 2026-08-06 02:40:34 on dist-test-slave-tfpf"
server_key: "3f28462303a41addb47f71c363596fd4"
server_key_iv: "b2d76838276749e808dc836c2c685304"
server_key_version: "encryptionkey@0"
I20260806 02:40:34.185874 977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:34.211192 977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:34.213933 977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:34.214131 977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:34.214672 977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:34.215456 977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:34.215545 977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:34.215628 977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:34.215677 977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:34.224072 977 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:46839
I20260806 02:40:34.224102 1177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:46839 every 8 connection(s)
I20260806 02:40:34.224928 977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:34.234920 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 977
I20260806 02:40:34.235070 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:34.235442 29436 external_mini_cluster.cc:1428] Setting key 15026c09298e30f79e555be9497345fe
I20260806 02:40:34.238632 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:34.226810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:43977 (local address 127.28.191.62:35707)
0806 02:40:34.226936 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:34.226940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:34.227767 (+ 827us) server_negotiation.cc:407] Connection header received
0806 02:40:34.229079 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:34.229083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:34.229134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:34.229233 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:34.230305 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.230822 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.231928 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.232172 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.232292 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:34.234682 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:34.234687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:34.234690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:34.234721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:34.236441 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:34.237149 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:34.237152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:34.237154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:34.237241 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:34.237641 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:34.237647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:34.237652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:34.237892 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:34.237984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:34.238423 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 02:40:34.238516 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:40:34.239408 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:34.227159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35707 (local address 127.28.191.3:43977)
0806 02:40:34.227605 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:34.227658 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:34.228842 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:34.229390 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:34.229397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:34.229721 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:34.230149 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:34.230162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.230956 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.230960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:34.231785 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:34.231795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:34.232326 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:34.232330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:34.232500 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:34.233052 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:34.233090 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:34.234502 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:34.236585 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:34.236593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:34.236606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:34.237032 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:34.237341 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:34.237344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:34.237346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:34.237533 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:34.238011 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:34.238018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:34.238223 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 02:40:34.238372 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:40:34.238611 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260806 02:40:34.240465 1178 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35707
I20260806 02:40:34.240697 1178 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:34.241221 1178 heartbeater.cc:511] Master 127.28.191.62:35707 requested a full tablet report, sending...
I20260806 02:40:34.242218 491 ts_manager.cc:194] Registered new tserver with Master: 8cd06ff9259a4ebebd516044ff8a2aca (127.28.191.3:46839)
I20260806 02:40:34.242770 491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:43977
I20260806 02:40:34.249569 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:40:34.264525 549 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:34.263032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.28.191.62:35707)
0806 02:40:34.263207 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:34.263211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:34.263345 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:40:34.263510 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:34.263513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:34.263558 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:34.263646 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:34.264380 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:40:34.267277 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 581
I20260806 02:40:34.282879 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 778
I20260806 02:40:34.292951 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 977
I20260806 02:40:34.303459 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 447
2026-08-06T02:40:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3239 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[1193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[1193](info): set up 2 sockets
Aug 06 02:40:34 dist-test-slave-tfpf krb5kdc[1193](info): commencing operation
krb5kdc: starting...
W20260806 02:40:35.525776 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:35.659873 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35117
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:35.769018 1207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:35.769272 1207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:35.769309 1207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:35.773102 1207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:35.773145 1207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:35.773164 1207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:35.773180 1207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:35.773195 1207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:35.778213 1207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:35.779143 1207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:35.780085 1207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:35.788439 1218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:35.788523 1207 server_base.cc:1034] running on GCE node
W20260806 02:40:35.788492 1219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:35.788654 1221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:35.789306 1207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:35.790460 1207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:35.791661 1207 hybrid_clock.cc:648] HybridClock initialized: now 1785984035791639 us; error 46 us; skew 500 ppm
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:35.794423 1207 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:35.795326 1207 webserver.cc:460] Webserver started at http://127.28.191.62:46285/ using document root <none> and password file <none>
I20260806 02:40:35.795908 1207 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:35.795969 1207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:35.796134 1207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:35.798148 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "34599f57e6234f7991f98416be55c445"
format_stamp: "Formatted at 2026-08-06 02:40:35 on dist-test-slave-tfpf"
server_key: "d519de8e844b215dd27a57c05ccb5612"
server_key_iv: "f8059d98b3294a77dc20bd1511af3011"
server_key_version: "encryptionkey@0"
I20260806 02:40:35.798662 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "34599f57e6234f7991f98416be55c445"
format_stamp: "Formatted at 2026-08-06 02:40:35 on dist-test-slave-tfpf"
server_key: "d519de8e844b215dd27a57c05ccb5612"
server_key_iv: "f8059d98b3294a77dc20bd1511af3011"
server_key_version: "encryptionkey@0"
I20260806 02:40:35.802243 1207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:35.804507 1233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:35.805828 1207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:35.805923 1207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "34599f57e6234f7991f98416be55c445"
format_stamp: "Formatted at 2026-08-06 02:40:35 on dist-test-slave-tfpf"
server_key: "d519de8e844b215dd27a57c05ccb5612"
server_key_iv: "f8059d98b3294a77dc20bd1511af3011"
server_key_version: "encryptionkey@0"
I20260806 02:40:35.806003 1207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:35.824302 1207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:35.827191 1207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:35.827342 1207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:35.834338 1207 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35117
I20260806 02:40:35.834700 1305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35117 every 8 connection(s)
I20260806 02:40:35.835364 1207 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:35.836187 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 1207
I20260806 02:40:35.836323 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:35.836591 29436 external_mini_cluster.cc:1428] Setting key ff33f4a4ae610b77f8507dea76e17c38
I20260806 02:40:35.838888 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:35.845836 1306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445: Bootstrap starting.
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:35.848628 1306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:35.849634 1306 log.cc:824] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:35.851590 1306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445: No bootstrap required, opened a new log
I20260806 02:40:35.851715 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:35.838025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.28.191.62:35117)
0806 02:40:35.838458 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:35.838478 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:35.838531 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:40:35.839382 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:35.839405 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:35.839815 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:35.840188 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:35.840903 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:35.842443 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:35.843292 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:35.844012 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:35.844171 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:35.846678 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:35.846685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:35.846699 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:35.846728 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:35.848762 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:35.849293 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:35.849299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:35.849304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:35.849459 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:35.849734 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:35.849739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:35.849741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:35.850224 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:35.850433 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:35.850747 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:40:35.851015 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260806 02:40:35.854642 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:40:35.855640 1306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } }
I20260806 02:40:35.855882 1306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:35.855974 1306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34599f57e6234f7991f98416be55c445, State: Initialized, Role: FOLLOWER
I20260806 02:40:35.856431 1306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } }
I20260806 02:40:35.856588 1306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:35.856674 1306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:35.856792 1306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:35.857723 1306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } }
I20260806 02:40:35.858067 1306 leader_election.cc:302] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34599f57e6234f7991f98416be55c445; no voters:
I20260806 02:40:35.858403 1306 leader_election.cc:288] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:35.858520 1316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:35.858748 1316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [term 1 LEADER]: Becoming Leader. State: Replica: 34599f57e6234f7991f98416be55c445, State: Running, Role: LEADER
I20260806 02:40:35.859133 1316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } }
I20260806 02:40:35.859669 1306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:35.860895 1318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34599f57e6234f7991f98416be55c445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } } }
I20260806 02:40:35.861133 1318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:35.861438 1318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34599f57e6234f7991f98416be55c445. Latest consensus state: current_term: 1 leader_uuid: "34599f57e6234f7991f98416be55c445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34599f57e6234f7991f98416be55c445" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35117 } } }
I20260806 02:40:35.861595 1318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:35.862944 1332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:35.865831 1332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:35.871934 1332 catalog_manager.cc:1347] Generated new cluster ID: bfb2bbb846de436d93126fd08c261de4
I20260806 02:40:35.871995 1332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:35.875026 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:35.869551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.28.191.62:35117)
0806 02:40:35.869671 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:35.869676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:35.869908 (+ 232us) server_negotiation.cc:407] Connection header received
0806 02:40:35.870048 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:35.870053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:35.870102 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:35.870195 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:35.870859 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:35.871382 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:35.872539 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:35.872926 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:35.873047 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:35.874012 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:35.874017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:35.874022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:35.874540 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:35.874577 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:35.874598 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:35.874694 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:35.874814 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:40:35.874877 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:40:35.878863 1332 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:35.879724 1332 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:35.890103 1332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34599f57e6234f7991f98416be55c445: Generated new TSK 0
I20260806 02:40:35.890729 1332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:40:35.902287 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 1207
2026-08-06T02:40:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1605 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1351](info): set up 2 sockets
Aug 06 02:40:35 dist-test-slave-tfpf krb5kdc[1351](info): commencing operation
krb5kdc: starting...
W20260806 02:40:37.110425 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:37.234174 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36151
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:37.343997 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:37.344229 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:37.344265 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:37.347987 1368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:37.348030 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:37.348049 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:37.348064 1368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:37.348078 1368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:37.353022 1368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:37.353928 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:37.354811 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:37.363268 1378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:37.363276 1375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:37.363467 1368 server_base.cc:1034] running on GCE node
W20260806 02:40:37.363276 1376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:37.364136 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:37.365197 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:37.366361 1368 hybrid_clock.cc:648] HybridClock initialized: now 1785984037366350 us; error 27 us; skew 500 ppm
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:37.369122 1368 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:37.369966 1368 webserver.cc:460] Webserver started at http://127.28.191.62:34211/ using document root <none> and password file <none>
I20260806 02:40:37.370507 1368 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:37.370613 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:37.370839 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:37.372794 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "ecab4fa9a65a47aa9e4085a4f3507af6"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "ab291340701f49b7accac36a20fef7b6"
server_key_iv: "7eb809907da56e72873a0380d720d570"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.373395 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "ecab4fa9a65a47aa9e4085a4f3507af6"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "ab291340701f49b7accac36a20fef7b6"
server_key_iv: "7eb809907da56e72873a0380d720d570"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.376914 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:37.379235 1385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.380393 1368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:37.380553 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "ecab4fa9a65a47aa9e4085a4f3507af6"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "ab291340701f49b7accac36a20fef7b6"
server_key_iv: "7eb809907da56e72873a0380d720d570"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.380666 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:37.394277 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:37.397148 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:37.397351 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:37.403936 1368 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36151
I20260806 02:40:37.404239 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36151 every 8 connection(s)
I20260806 02:40:37.404863 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:37.407528 1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:37.410494 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 1368
I20260806 02:40:37.410665 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:37.410931 29436 external_mini_cluster.cc:1428] Setting key 8103396a5a35639d86e0e9400ad4dd9c
I20260806 02:40:37.413650 1462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6: Bootstrap starting.
I20260806 02:40:37.416194 1462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:37.416949 1462 log.cc:824] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:37.419023 1462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6: No bootstrap required, opened a new log
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:37.423192 1462 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } }
I20260806 02:40:37.423458 1462 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:37.423556 1462 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecab4fa9a65a47aa9e4085a4f3507af6, State: Initialized, Role: FOLLOWER
I20260806 02:40:37.424080 1462 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } }
I20260806 02:40:37.424254 1462 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:37.424343 1462 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:37.424443 1462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:37.425371 1462 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } }
I20260806 02:40:37.425742 1462 leader_election.cc:302] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecab4fa9a65a47aa9e4085a4f3507af6; no voters:
I20260806 02:40:37.426086 1462 leader_election.cc:288] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:37.426146 1468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:37.426363 1468 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [term 1 LEADER]: Becoming Leader. State: Replica: ecab4fa9a65a47aa9e4085a4f3507af6, State: Running, Role: LEADER
I20260806 02:40:37.426668 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:37.412168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.28.191.62:36151)
0806 02:40:37.412590 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:37.412610 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:37.412653 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:40:37.413453 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:37.413475 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:37.413900 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:37.414233 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:37.414893 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.416222 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.417167 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.417734 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.417893 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:37.420578 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:37.420586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:37.420603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:37.420633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:37.423295 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.423833 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.423839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.423844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.424011 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.424343 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.424346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.424348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.424816 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:37.425036 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:37.425367 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 02:40:37.425674 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260806 02:40:37.426703 1468 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } }
I20260806 02:40:37.427265 1462 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:37.427795 1468 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecab4fa9a65a47aa9e4085a4f3507af6. Latest consensus state: current_term: 1 leader_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } } }
I20260806 02:40:37.427964 1468 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:37.428256 1470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecab4fa9a65a47aa9e4085a4f3507af6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36151 } } }
I20260806 02:40:37.428356 1470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:37.428718 1478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:37.431749 1478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:37.437639 1478 catalog_manager.cc:1347] Generated new cluster ID: a99409f5fb304464aa7564d0238731f9
I20260806 02:40:37.437754 1478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:37.456249 1478 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:37.457489 1478 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:37.467365 1478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecab4fa9a65a47aa9e4085a4f3507af6: Generated new TSK 0
I20260806 02:40:37.468037 1478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:37.504120 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:37.635396 1498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:37.635627 1498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:37.635666 1498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:37.639786 1498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:37.639850 1498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:37.639954 1498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:37.645186 1498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:37.646195 1498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:37.647214 1498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:37.656684 1516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:37.656661 1518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:37.656651 1515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:37.657027 1498 server_base.cc:1034] running on GCE node
I20260806 02:40:37.657469 1498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:37.658097 1498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:37.659292 1498 hybrid_clock.cc:648] HybridClock initialized: now 1785984037659281 us; error 36 us; skew 500 ppm
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:37.662106 1498 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:37.663271 1498 webserver.cc:460] Webserver started at http://127.28.191.1:43093/ using document root <none> and password file <none>
I20260806 02:40:37.664065 1498 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:37.664199 1498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:37.664469 1498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:37.666841 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "bb44a61aa8204266b8094a5fe59f41e0"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "26efb954a10224657ef07c5c13ffeb81"
server_key_iv: "843121ffeef70db632cdb7cb726e66ab"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.667419 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "bb44a61aa8204266b8094a5fe59f41e0"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "26efb954a10224657ef07c5c13ffeb81"
server_key_iv: "843121ffeef70db632cdb7cb726e66ab"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.671037 1498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 02:40:37.673321 1529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.674475 1498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:37.674620 1498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "bb44a61aa8204266b8094a5fe59f41e0"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "26efb954a10224657ef07c5c13ffeb81"
server_key_iv: "843121ffeef70db632cdb7cb726e66ab"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.674731 1498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:37.686931 1498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:37.689595 1498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:37.689803 1498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:37.690332 1498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:37.691098 1498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:37.691193 1498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.691260 1498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:37.691321 1498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.700131 1498 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:45391
I20260806 02:40:37.700178 1683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:45391 every 8 connection(s)
I20260806 02:40:37.700965 1498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:37.701822 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 1498
I20260806 02:40:37.701953 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:37.702260 29436 external_mini_cluster.cc:1428] Setting key 0cc5937e8b280e4f54da567639d5c1ab
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:40:37.718143 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:37.702882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:48695 (local address 127.28.191.62:36151)
0806 02:40:37.703022 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:37.703027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:37.703916 (+ 889us) server_negotiation.cc:407] Connection header received
0806 02:40:37.706833 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:37.706840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:37.706903 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:37.707004 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:37.708394 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.708918 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.710756 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.710984 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.711065 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:37.713907 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:37.713912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:37.713915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:37.713944 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:37.715413 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.715999 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.716004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.716007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.716229 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.716641 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.716646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.716651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.716963 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:37.717244 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:37.717894 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 02:40:37.718024 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:40:37.719085 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:37.703243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36151 (local address 127.28.191.1:48695)
0806 02:40:37.703761 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:37.703816 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:37.706630 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:37.707163 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:37.707169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:37.707561 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:37.708234 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:37.708257 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.709081 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.709088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:37.710625 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:37.710641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.711147 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.711154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:37.711383 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:37.711943 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:37.711969 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:37.713717 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:37.715522 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:37.715527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:37.715538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:37.715845 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:37.716332 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:37.716334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:37.716335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:37.716504 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:37.717241 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:37.717248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:37.717607 (+ 359us) client_negotiation.cc:769] Sending connection context
0806 02:40:37.717831 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 02:40:37.718188 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":133,"threads_started":1}
I20260806 02:40:37.720325 1685 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36151
I20260806 02:40:37.720588 1685 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:37.721253 1685 heartbeater.cc:511] Master 127.28.191.62:36151 requested a full tablet report, sending...
I20260806 02:40:37.722913 1414 ts_manager.cc:194] Registered new tserver with Master: bb44a61aa8204266b8094a5fe59f41e0 (127.28.191.1:45391)
I20260806 02:40:37.724093 1414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:48695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:37.733873 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:37.844363 1693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:37.844587 1693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:37.844625 1693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:37.848419 1693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:37.848469 1693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:37.848517 1693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:37.853610 1693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:37.854501 1693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:37.855393 1693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:37.864872 1702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:37.864954 1693 server_base.cc:1034] running on GCE node
W20260806 02:40:37.864861 1705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:37.864872 1699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:37.865481 1693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:37.866076 1693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:37.867265 1693 hybrid_clock.cc:648] HybridClock initialized: now 1785984037867233 us; error 41 us; skew 500 ppm
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:37.870362 1693 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:37.871147 1693 webserver.cc:460] Webserver started at http://127.28.191.2:37563/ using document root <none> and password file <none>
I20260806 02:40:37.871659 1693 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:37.871721 1693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:37.871882 1693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:37.873942 1693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "c95665bd43d64634abb50e6ad1dca2aa"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "4901408b19b8bd083966747d0c0c8a4a"
server_key_iv: "c4f85c70e47c44b328f6026a2b68e391"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.874503 1693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "c95665bd43d64634abb50e6ad1dca2aa"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "4901408b19b8bd083966747d0c0c8a4a"
server_key_iv: "c4f85c70e47c44b328f6026a2b68e391"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.878340 1693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:40:37.880972 1713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.882236 1693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:37.882396 1693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "c95665bd43d64634abb50e6ad1dca2aa"
format_stamp: "Formatted at 2026-08-06 02:40:37 on dist-test-slave-tfpf"
server_key: "4901408b19b8bd083966747d0c0c8a4a"
server_key_iv: "c4f85c70e47c44b328f6026a2b68e391"
server_key_version: "encryptionkey@0"
I20260806 02:40:37.882534 1693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:37.901266 1693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:37.904820 1693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:37.905083 1693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:37.905730 1693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:37.906600 1693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:37.906682 1693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.906754 1693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:37.906807 1693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:37.916296 1693 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:39101
I20260806 02:40:37.916405 1886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:39101 every 8 connection(s)
I20260806 02:40:37.917297 1693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:37.920580 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 1693
I20260806 02:40:37.920755 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:37.921195 29436 external_mini_cluster.cc:1428] Setting key 632b6aa133929722134c5e572626a060
Aug 06 02:40:37 dist-test-slave-tfpf krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:40:37.932752 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:37.919123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:48931 (local address 127.28.191.62:36151)
0806 02:40:37.919295 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:37.919300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:37.920080 (+ 780us) server_negotiation.cc:407] Connection header received
0806 02:40:37.921925 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:37.921928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:37.921997 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:37.922085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:37.923685 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.924181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.925685 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.925926 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.926055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:37.928865 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:37.928869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:37.928872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:37.928908 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:37.930437 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.931005 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.931013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.931016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.931102 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:37.931487 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:37.931491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:37.931493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:37.931696 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:37.931816 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:37.932429 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 02:40:37.932564 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:40:37.933787 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:37.919456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36151 (local address 127.28.191.2:48931)
0806 02:40:37.919898 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:37.919945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:37.921681 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:37.922246 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:37.922271 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:37.922818 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:37.923508 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:37.923544 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.924315 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.924322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:37.925533 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:37.925543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:37.926088 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:37.926093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:37.926268 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:37.926929 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:37.926947 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:37.928676 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:37.930564 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:37.930571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:37.930584 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:37.930858 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:37.931218 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:37.931221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:37.931223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:37.931373 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:37.931824 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:37.931834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:37.932145 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 02:40:37.932375 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 02:40:37.932698 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
I20260806 02:40:37.935086 1887 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36151
I20260806 02:40:37.935302 1887 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:37.935745 1887 heartbeater.cc:511] Master 127.28.191.62:36151 requested a full tablet report, sending...
I20260806 02:40:37.936771 1414 ts_manager.cc:194] Registered new tserver with Master: c95665bd43d64634abb50e6ad1dca2aa (127.28.191.2:39101)
I20260806 02:40:37.937412 1414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:48931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:37.949741 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:38.059899 1892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:38.060173 1892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:38.060249 1892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:38.064297 1892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:38.064378 1892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:38.064462 1892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:38.069581 1892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:38.070557 1892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:38.071561 1892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:38.080914 1903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:38.080955 1892 server_base.cc:1034] running on GCE node
W20260806 02:40:38.080909 1899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:38.080886 1898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:38.081527 1892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:38.082101 1892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:38.083288 1892 hybrid_clock.cc:648] HybridClock initialized: now 1785984038083270 us; error 29 us; skew 500 ppm
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984038, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:38.086150 1892 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:38.087010 1892 webserver.cc:460] Webserver started at http://127.28.191.3:40399/ using document root <none> and password file <none>
I20260806 02:40:38.087530 1892 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:38.087589 1892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:38.087807 1892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:38.089741 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "21dce33e22074e4cbb56dce7fc956a58"
format_stamp: "Formatted at 2026-08-06 02:40:38 on dist-test-slave-tfpf"
server_key: "2a901d63157dcfdb6177adc5bc6ba45f"
server_key_iv: "d7bc12958917fe5435f878e389ea6208"
server_key_version: "encryptionkey@0"
I20260806 02:40:38.090301 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "21dce33e22074e4cbb56dce7fc956a58"
format_stamp: "Formatted at 2026-08-06 02:40:38 on dist-test-slave-tfpf"
server_key: "2a901d63157dcfdb6177adc5bc6ba45f"
server_key_iv: "d7bc12958917fe5435f878e389ea6208"
server_key_version: "encryptionkey@0"
I20260806 02:40:38.093839 1892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:40:38.096159 1915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:38.097348 1892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:38.097481 1892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "21dce33e22074e4cbb56dce7fc956a58"
format_stamp: "Formatted at 2026-08-06 02:40:38 on dist-test-slave-tfpf"
server_key: "2a901d63157dcfdb6177adc5bc6ba45f"
server_key_iv: "d7bc12958917fe5435f878e389ea6208"
server_key_version: "encryptionkey@0"
I20260806 02:40:38.097585 1892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:38.113940 1892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:38.116632 1892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:38.116899 1892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:38.117529 1892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:38.118306 1892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:38.118379 1892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:38.118443 1892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:38.118500 1892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:38.127372 1892 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:43525
I20260806 02:40:38.127403 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:43525 every 8 connection(s)
I20260806 02:40:38.128330 1892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:38.136061 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 1892
I20260806 02:40:38.136188 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:38.136502 29436 external_mini_cluster.cc:1428] Setting key 00ba37493f57e5f14b5d87ef96418e75
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984038, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:38.141547 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:38.130039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:47197 (local address 127.28.191.62:36151)
0806 02:40:38.130206 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:38.130211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:38.130951 (+ 740us) server_negotiation.cc:407] Connection header received
0806 02:40:38.132194 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:38.132196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:38.132245 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:38.132335 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:38.133482 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.133980 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.135107 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.135320 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.135398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:38.138036 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:38.138039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:38.138042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:38.138070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:38.139685 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:38.140179 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:38.140181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:38.140183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:38.140272 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:38.140586 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:38.140588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:38.140589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:38.140745 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:38.140820 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:38.141345 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 02:40:38.141437 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:40:38.142299 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:38.130384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36151 (local address 127.28.191.3:47197)
0806 02:40:38.130800 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:38.130839 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:38.132011 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:38.132486 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:38.132492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:38.132820 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:38.133340 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:38.133354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.134120 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.134123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:38.134994 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:38.135004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.135442 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.135445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:38.135586 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:38.136076 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:38.136099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:38.137862 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:38.139799 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:38.139804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:38.139814 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:38.140087 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:38.140357 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:38.140359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:38.140360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:38.140503 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:38.140840 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:38.140844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:38.141062 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:40:38.141298 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 02:40:38.141538 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260806 02:40:38.143368 2071 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36151
I20260806 02:40:38.143602 2071 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:38.144049 2071 heartbeater.cc:511] Master 127.28.191.62:36151 requested a full tablet report, sending...
I20260806 02:40:38.145133 1414 ts_manager.cc:194] Registered new tserver with Master: 21dce33e22074e4cbb56dce7fc956a58 (127.28.191.3:43525)
I20260806 02:40:38.145699 1414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:47197
I20260806 02:40:38.150892 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:38.177697 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:38.169048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.28.191.62:36151)
0806 02:40:38.169195 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:38.169199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:38.169380 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:40:38.169490 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:38.169492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:38.169532 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:38.169611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:38.170394 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.170887 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.171692 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:38.171889 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:38.171961 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:38.174600 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:38.174603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:38.174605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:38.174627 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:38.175916 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:38.176354 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:38.176359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:38.176361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:38.176449 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:38.176767 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:38.176771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:38.176773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:38.176986 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:38.177168 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:38.177413 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 02:40:38.177532 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 02:40:38.178130 1414 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39218
I20260806 02:40:38.178960 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 1498
I20260806 02:40:38.193565 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 1693
I20260806 02:40:38.200448 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 1892
I20260806 02:40:38.212591 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 1368
2026-08-06T02:40:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2305 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[2087](info): set up 2 sockets
Aug 06 02:40:38 dist-test-slave-tfpf krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260806 02:40:39.400127 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:39 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:39.518420 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38637
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:38637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:39.625794 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:39.626034 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:39.626072 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:39.629722 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:39.629765 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:39.629782 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:39.629797 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:39.629856 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:39.634688 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:38637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:39.635643 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:39.636574 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:39.644917 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:39.644917 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:39.644999 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:39.645094 2105 server_base.cc:1034] running on GCE node
I20260806 02:40:39.645748 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:39.646767 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:39.647957 2105 hybrid_clock.cc:648] HybridClock initialized: now 1785984039647936 us; error 35 us; skew 500 ppm
Aug 06 02:40:39 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984039, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:39.654608 2105 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:39.655579 2105 webserver.cc:460] Webserver started at http://127.28.191.62:43177/ using document root <none> and password file <none>
I20260806 02:40:39.656109 2105 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:39.656169 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:39.656342 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:39.658438 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "5d35190d7ee230313f0cf5c5642f5ff5"
server_key_iv: "8cc21aca51246277d9491c5b44757d5b"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.658952 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "5d35190d7ee230313f0cf5c5642f5ff5"
server_key_iv: "8cc21aca51246277d9491c5b44757d5b"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.662546 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:39.664808 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:39.666033 2105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:39.666155 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "5d35190d7ee230313f0cf5c5642f5ff5"
server_key_iv: "8cc21aca51246277d9491c5b44757d5b"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.666247 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:39.677977 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:39.680656 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:39.680802 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:39.687788 2105 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:38637
I20260806 02:40:39.687799 2194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:38637 every 8 connection(s)
I20260806 02:40:39.688753 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:39.691512 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:39.694363 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2105
I20260806 02:40:39.694491 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:39.694761 29436 external_mini_cluster.cc:1428] Setting key 771f332754c81a1b1526dfef4e0575df
I20260806 02:40:39.697535 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22: Bootstrap starting.
I20260806 02:40:39.700376 2195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:39.701213 2195 log.cc:824] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:39.703119 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22: No bootstrap required, opened a new log
Aug 06 02:40:39 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:39.706441 2195 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } }
I20260806 02:40:39.706699 2195 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:39.706770 2195 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b8e9b9ef0f46fdacb99cf4367f6a22, State: Initialized, Role: FOLLOWER
I20260806 02:40:39.707193 2195 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } }
I20260806 02:40:39.707345 2195 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:39.707432 2195 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:39.707563 2195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:39.708377 2195 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } }
I20260806 02:40:39.708943 2195 leader_election.cc:302] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b8e9b9ef0f46fdacb99cf4367f6a22; no voters:
I20260806 02:40:39.709337 2195 leader_election.cc:288] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:39.709458 2201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:39.709717 2201 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [term 1 LEADER]: Becoming Leader. State: Replica: a4b8e9b9ef0f46fdacb99cf4367f6a22, State: Running, Role: LEADER
I20260806 02:40:39.710147 2201 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } }
I20260806 02:40:39.710753 2195 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:39.711534 2201 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b8e9b9ef0f46fdacb99cf4367f6a22. Latest consensus state: current_term: 1 leader_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } } }
I20260806 02:40:39.711782 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:39.696108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.28.191.62:38637)
0806 02:40:39.696534 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:39.696564 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:39.696604 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:40:39.697438 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:39.697470 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:39.697876 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:39.698213 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:39.698910 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.700416 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.701472 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.701922 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.702122 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:39.705174 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:39.705180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:39.705193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:39.705221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:39.707946 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:39.708519 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:39.708526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:39.708534 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:39.708651 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:39.709012 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:39.709014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:39.709015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:39.709798 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:39.710077 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:39.710417 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:40:39.710761 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260806 02:40:39.712309 2201 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:39.712770 2221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:39.712183 2205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b8e9b9ef0f46fdacb99cf4367f6a22" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38637 } } }
I20260806 02:40:39.713896 2205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:39.716845 2221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:39.722669 2221 catalog_manager.cc:1347] Generated new cluster ID: f3cef8756d18488ea017948652290569
I20260806 02:40:39.722734 2221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:39.732221 2221 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:39.733426 2221 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:39.742293 2221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b8e9b9ef0f46fdacb99cf4367f6a22: Generated new TSK 0
I20260806 02:40:39.742939 2221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:39.774452 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:39.884013 2232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:39.884290 2232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:39.884361 2232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:39.888267 2232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:39.888352 2232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:39.888438 2232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:39.893662 2232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:39.894624 2232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:39.895574 2232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:40:39.905138 2232 server_base.cc:1034] running on GCE node
W20260806 02:40:39.905098 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:39.905040 2240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:39.905084 2241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:39.905761 2232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:39.906355 2232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:39.907548 2232 hybrid_clock.cc:648] HybridClock initialized: now 1785984039907539 us; error 30 us; skew 500 ppm
Aug 06 02:40:39 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984039, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:39.910456 2232 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:39.911262 2232 webserver.cc:460] Webserver started at http://127.28.191.1:33265/ using document root <none> and password file <none>
I20260806 02:40:39.911808 2232 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:39.911867 2232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:39.912051 2232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:39.914074 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "f992f3ad033d4b97b3fc5612d86d96e6"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "a059d0198eca066e88ea0386edc51810"
server_key_iv: "934227a4259237a180c28d27731b6682"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.914561 2232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "f992f3ad033d4b97b3fc5612d86d96e6"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "a059d0198eca066e88ea0386edc51810"
server_key_iv: "934227a4259237a180c28d27731b6682"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.917917 2232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:39.920051 2253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:39.921320 2232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:39.921433 2232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f992f3ad033d4b97b3fc5612d86d96e6"
format_stamp: "Formatted at 2026-08-06 02:40:39 on dist-test-slave-tfpf"
server_key: "a059d0198eca066e88ea0386edc51810"
server_key_iv: "934227a4259237a180c28d27731b6682"
server_key_version: "encryptionkey@0"
I20260806 02:40:39.921527 2232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:39.949488 2232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:39.952695 2232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:39.952900 2232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:39.953539 2232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:39.954362 2232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:39.954408 2232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:39.954473 2232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:39.954502 2232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:39.963438 2232 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:46767
I20260806 02:40:39.963465 2410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:46767 every 8 connection(s)
I20260806 02:40:39.964253 2232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:39.971279 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2232
I20260806 02:40:39.971467 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:39.971779 29436 external_mini_cluster.cc:1428] Setting key 8a73fa33a4e02c44a2c029acc7ef323a
Aug 06 02:40:39 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984039, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:39.978500 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:39.966026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:47805 (local address 127.28.191.62:38637)
0806 02:40:39.966154 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:39.966158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:39.966921 (+ 763us) server_negotiation.cc:407] Connection header received
0806 02:40:39.968296 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:39.968299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:39.968356 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:39.968462 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:39.969636 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.970223 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.971422 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.971711 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.971827 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:39.974487 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:39.974491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:39.974493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:39.974517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:39.976245 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:39.976754 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:39.976777 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:39.976782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:39.976877 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:39.977270 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:39.977276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:39.977280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:39.977527 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:39.977646 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:39.978110 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 02:40:39.978256 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:40:39.979029 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:39.966346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38637 (local address 127.28.191.1:47805)
0806 02:40:39.966780 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:39.966820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:39.968088 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:39.968634 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:39.968641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:39.968958 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:39.969486 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:39.969499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.970356 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.970360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:39.971212 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:39.971222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:39.971873 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:39.971878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:39.972045 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:39.972635 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:39.972651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:39.974328 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:39.976370 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:39.976375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:39.976385 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:39.976634 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:39.976975 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:39.976977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:39.976978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:39.977172 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:39.977643 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:39.977648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:39.977854 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 02:40:39.978006 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 02:40:39.978234 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":126,"threads_started":1}
I20260806 02:40:39.980255 2411 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38637
I20260806 02:40:39.980491 2411 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:39.980953 2411 heartbeater.cc:511] Master 127.28.191.62:38637 requested a full tablet report, sending...
I20260806 02:40:39.982381 2146 ts_manager.cc:194] Registered new tserver with Master: f992f3ad033d4b97b3fc5612d86d96e6 (127.28.191.1:46767)
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:39.983532 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:47805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:40.000133 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:40.111539 2417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:40.111814 2417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:40.111912 2417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:40.115993 2417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:40.116072 2417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:40.116151 2417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:40.121284 2417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:40.122350 2417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:40.123342 2417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:40.132704 2427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:40.132702 2425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:40.132695 2424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:40.133033 2417 server_base.cc:1034] running on GCE node
I20260806 02:40:40.133491 2417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:40.134073 2417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:40.135254 2417 hybrid_clock.cc:648] HybridClock initialized: now 1785984040135238 us; error 28 us; skew 500 ppm
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:40.138223 2417 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:40.139132 2417 webserver.cc:460] Webserver started at http://127.28.191.2:38091/ using document root <none> and password file <none>
I20260806 02:40:40.139691 2417 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:40.139837 2417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:40.140111 2417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:40.142141 2417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "74e5c8a633264043b9bc02322c115db7"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "704605b0c17f644997c1b7f91c6a1242"
server_key_iv: "c07a192f406574a9e458d8ab13a2dcda"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.142776 2417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "74e5c8a633264043b9bc02322c115db7"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "704605b0c17f644997c1b7f91c6a1242"
server_key_iv: "c07a192f406574a9e458d8ab13a2dcda"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.147025 2417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:40:40.149533 2436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.150722 2417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:40.150866 2417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "74e5c8a633264043b9bc02322c115db7"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "704605b0c17f644997c1b7f91c6a1242"
server_key_iv: "c07a192f406574a9e458d8ab13a2dcda"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.150977 2417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:40.167776 2417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:40.171669 2417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:40.171885 2417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:40.172683 2417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:40.173731 2417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:40.173787 2417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.173858 2417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:40.173895 2417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.183405 2417 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:34845
I20260806 02:40:40.183490 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:34845 every 8 connection(s)
I20260806 02:40:40.184322 2417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:40.187206 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2417
I20260806 02:40:40.187336 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:40.187677 29436 external_mini_cluster.cc:1428] Setting key 5a6c2f9aeb554e63bdeb9dd336403868
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:40:40.199308 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.186154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:41961 (local address 127.28.191.62:38637)
0806 02:40:40.186302 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:40.186307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:40.187236 (+ 929us) server_negotiation.cc:407] Connection header received
0806 02:40:40.188994 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:40.188998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:40.189094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:40.189188 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:40.190794 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.191340 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.192486 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.192714 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.192825 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.195657 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:40.195660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:40.195662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:40.195686 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:40.197270 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.197793 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.197798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.197800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.197884 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.198225 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.198230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.198232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.198410 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:40.198521 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:40.199015 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 02:40:40.199143 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:40:40.200157 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.186560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38637 (local address 127.28.191.2:41961)
0806 02:40:40.187045 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:40.187106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:40.188787 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:40.189398 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:40.189408 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:40.189936 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:40.190635 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:40.190658 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.191468 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.191480 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:40.192343 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:40.192353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.192864 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.192869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:40.193119 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.193732 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:40.193758 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:40.195475 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:40.197390 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:40.197396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:40.197410 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:40.197686 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:40.197980 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:40.197984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:40.197986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:40.198128 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:40.198520 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:40.198527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:40.198784 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 02:40:40.198969 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:40:40.199266 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
I20260806 02:40:40.201376 2607 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38637
I20260806 02:40:40.201625 2607 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:40.202099 2607 heartbeater.cc:511] Master 127.28.191.62:38637 requested a full tablet report, sending...
I20260806 02:40:40.203157 2146 ts_manager.cc:194] Registered new tserver with Master: 74e5c8a633264043b9bc02322c115db7 (127.28.191.2:34845)
I20260806 02:40:40.203711 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:41961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:40.215638 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:40.327081 2614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:40.327320 2614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:40.327359 2614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:40.331316 2614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:40.331373 2614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:40.331429 2614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:40.336683 2614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:40.337646 2614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:40.338621 2614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:40.348325 2621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:40.348330 2623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:40.348364 2620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:40.348773 2614 server_base.cc:1034] running on GCE node
I20260806 02:40:40.349216 2614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:40.349826 2614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:40.351030 2614 hybrid_clock.cc:648] HybridClock initialized: now 1785984040351011 us; error 36 us; skew 500 ppm
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:40.353916 2614 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:40.354904 2614 webserver.cc:460] Webserver started at http://127.28.191.3:38587/ using document root <none> and password file <none>
I20260806 02:40:40.355482 2614 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:40.355583 2614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:40.355829 2614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:40.357955 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "ff953e4eb8874a22a9f7ebc36799f9f2"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "b381e9e2eb43175b837644ae6bec7bce"
server_key_iv: "798c1b065a19f2d01445bcfc6caeb834"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.358582 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "ff953e4eb8874a22a9f7ebc36799f9f2"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "b381e9e2eb43175b837644ae6bec7bce"
server_key_iv: "798c1b065a19f2d01445bcfc6caeb834"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.362401 2614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:40:40.364926 2632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.366127 2614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:40.366256 2614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "ff953e4eb8874a22a9f7ebc36799f9f2"
format_stamp: "Formatted at 2026-08-06 02:40:40 on dist-test-slave-tfpf"
server_key: "b381e9e2eb43175b837644ae6bec7bce"
server_key_iv: "798c1b065a19f2d01445bcfc6caeb834"
server_key_version: "encryptionkey@0"
I20260806 02:40:40.366364 2614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:40.379637 2614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:40.382387 2614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:40.382609 2614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:40.383152 2614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:40.383950 2614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:40.384025 2614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.384095 2614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:40.384145 2614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:40.392639 2614 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:41683
I20260806 02:40:40.392665 2796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:41683 every 8 connection(s)
I20260806 02:40:40.393527 2614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:40.401901 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2614
I20260806 02:40:40.402068 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:40.402359 29436 external_mini_cluster.cc:1428] Setting key 99abc3c8c1693d71a95c6e8441c651e4
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:40.406656 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.395348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:38653 (local address 127.28.191.62:38637)
0806 02:40:40.395481 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:40.395485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:40.396209 (+ 724us) server_negotiation.cc:407] Connection header received
0806 02:40:40.397629 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:40.397632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:40.397691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:40.397801 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:40.398899 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.399403 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.400469 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.400693 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.400801 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.403354 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:40.403358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:40.403361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:40.403391 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:40.404936 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.405435 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.405438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.405439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.405515 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.405815 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.405817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.405818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.405976 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:40.406062 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:40.406453 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:40:40.406547 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:40:40.407352 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.395653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38637 (local address 127.28.191.3:38653)
0806 02:40:40.396068 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:40.396107 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:40.397413 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:40.397963 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:40.397969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:40.398302 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:40.398750 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:40.398762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.399526 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.399529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:40.400342 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:40.400351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.400836 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.400840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:40.400999 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.401575 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:40.401591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:40.403188 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:40.405036 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:40.405041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:40.405051 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:40.405349 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:40.405599 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:40.405601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:40.405602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:40.405723 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:40.406057 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:40.406062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:40.406259 (+ 197us) client_negotiation.cc:769] Sending connection context
0806 02:40:40.406393 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:40:40.406620 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260806 02:40:40.408342 2797 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38637
I20260806 02:40:40.408571 2797 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:40.409034 2797 heartbeater.cc:511] Master 127.28.191.62:38637 requested a full tablet report, sending...
I20260806 02:40:40.410094 2146 ts_manager.cc:194] Registered new tserver with Master: ff953e4eb8874a22a9f7ebc36799f9f2 (127.28.191.3:41683)
I20260806 02:40:40.410702 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:38653
I20260806 02:40:40.416040 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:40.443020 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.433719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.28.191.1:46767)
0806 02:40:40.434016 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:40.434031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:40.434070 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:40:40.434131 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:40.434136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:40.434327 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:40.434474 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:40.435205 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.435985 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.436788 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.437214 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.437298 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.439484 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:40.439494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:40.439506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:40.439530 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:40.441201 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.441599 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.441603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.441605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.441691 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.441928 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.441930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.441931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.442202 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:40.442375 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:40.442613 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:40:40.442790 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:40.453213 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:40.445847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.28.191.62:38637)
0806 02:40:40.446015 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:40.446019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:40.446033 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:40.446174 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:40.446181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:40.446222 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:40.446301 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:40.446877 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.447355 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.448097 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:40.448309 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:40.448415 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:40.450589 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:40.450592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:40.450594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:40.450615 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:40.451739 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.452104 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.452106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.452108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.452179 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:40.452400 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:40.452402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:40.452403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:40.452620 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:40.452721 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:40.452905 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:40:40.452996 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 02:40:40.453611 2146 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33966
I20260806 02:40:40.454231 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2232
I20260806 02:40:40.475106 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2417
I20260806 02:40:40.485553 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2614
I20260806 02:40:40.494613 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2105
2026-08-06T02:40:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2281 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2811](info): set up 2 sockets
Aug 06 02:40:40 dist-test-slave-tfpf krb5kdc[2811](info): commencing operation
krb5kdc: starting...
W20260806 02:40:41.709832 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:41 dist-test-slave-tfpf krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:41.834295 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33367
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:41.942534 2833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:41.942832 2833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:41.942902 2833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:41.946782 2833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:41.946856 2833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:41.946892 2833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:41.946950 2833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:41.946990 2833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:41.951745 2833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:41.952713 2833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:41.953680 2833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:41.961638 2846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:41.961663 2842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:41.961786 2833 server_base.cc:1034] running on GCE node
W20260806 02:40:41.961663 2843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:41.962453 2833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:41.963486 2833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:41.964684 2833 hybrid_clock.cc:648] HybridClock initialized: now 1785984041964659 us; error 45 us; skew 500 ppm
Aug 06 02:40:41 dist-test-slave-tfpf krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984041, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:41.967584 2833 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:41.968300 2833 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:41.968359 2833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:41.968510 2833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:41.970435 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "eb8b8376db454e779231ff3feba662bb"
format_stamp: "Formatted at 2026-08-06 02:40:41 on dist-test-slave-tfpf"
server_key: "80bfeb899bc3e54a676b017476d1c2a8"
server_key_iv: "f041b3793ab468efb4518d30a895155b"
server_key_version: "encryptionkey@0"
I20260806 02:40:41.970914 2833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "eb8b8376db454e779231ff3feba662bb"
format_stamp: "Formatted at 2026-08-06 02:40:41 on dist-test-slave-tfpf"
server_key: "80bfeb899bc3e54a676b017476d1c2a8"
server_key_iv: "f041b3793ab468efb4518d30a895155b"
server_key_version: "encryptionkey@0"
I20260806 02:40:41.974352 2833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:41.976562 2853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:41.977756 2833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:41.977880 2833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "eb8b8376db454e779231ff3feba662bb"
format_stamp: "Formatted at 2026-08-06 02:40:41 on dist-test-slave-tfpf"
server_key: "80bfeb899bc3e54a676b017476d1c2a8"
server_key_iv: "f041b3793ab468efb4518d30a895155b"
server_key_version: "encryptionkey@0"
I20260806 02:40:41.977985 2833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:41.990756 2833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:41.993415 2833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:41.993614 2833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:42.000736 2833 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:33367
I20260806 02:40:42.001050 2933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:33367 every 8 connection(s)
I20260806 02:40:42.001494 2833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:42.004343 2934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:42.010027 2934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: Bootstrap starting.
I20260806 02:40:42.010314 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2833
I20260806 02:40:42.010416 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:42.010625 29436 external_mini_cluster.cc:1428] Setting key aa95c1a3b1e9cf604d412b5e5cfbe882
I20260806 02:40:42.012452 2934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:42.013271 2934 log.cc:824] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:42.015342 2934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: No bootstrap required, opened a new log
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:42.019937 2934 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } }
I20260806 02:40:42.020231 2934 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:42.020313 2934 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8b8376db454e779231ff3feba662bb, State: Initialized, Role: FOLLOWER
I20260806 02:40:42.020825 2934 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } }
I20260806 02:40:42.020994 2934 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:42.021096 2934 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:42.021243 2934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:42.022408 2934 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } }
I20260806 02:40:42.022814 2934 leader_election.cc:302] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb8b8376db454e779231ff3feba662bb; no voters:
I20260806 02:40:42.023130 2934 leader_election.cc:288] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:42.023245 2945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:42.023463 2945 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [term 1 LEADER]: Becoming Leader. State: Replica: eb8b8376db454e779231ff3feba662bb, State: Running, Role: LEADER
I20260806 02:40:42.023880 2945 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } }
I20260806 02:40:42.024354 2934 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:42.025794 2946 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb8b8376db454e779231ff3feba662bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } } }
I20260806 02:40:42.026006 2946 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:42.026212 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.011965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.28.191.62:33367)
0806 02:40:42.012380 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.012396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.012432 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:40:42.013166 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.013184 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.013550 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.013927 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:42.014780 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.015765 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.017004 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.017522 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.017640 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.020519 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:42.020524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:42.020539 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:42.020572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:42.022841 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.023418 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.023429 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.023438 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.023637 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.023965 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.023968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.023969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.024438 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:42.024612 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.024936 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.025268 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
I20260806 02:40:42.026851 2947 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8b8376db454e779231ff3feba662bb. Latest consensus state: current_term: 1 leader_uuid: "eb8b8376db454e779231ff3feba662bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb8b8376db454e779231ff3feba662bb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 33367 } } }
I20260806 02:40:42.027024 2947 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:42.026938 2954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:42.030413 2954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:42.035717 2954 catalog_manager.cc:1347] Generated new cluster ID: 8fa93818c6194903b15ca23036c40ae5
I20260806 02:40:42.035773 2954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:42.060710 2954 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:42.061728 2954 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:42.068223 2954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: Generated new TSK 0
I20260806 02:40:42.068949 2954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:42.099735 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:42.210843 2969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:42.211071 2969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:42.211112 2969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:42.215026 2969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:42.215072 2969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:42.215167 2969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:42.219991 2969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:42.220921 2969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:42.221944 2969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:40:42.231612 2969 server_base.cc:1034] running on GCE node
W20260806 02:40:42.231557 2976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:42.231550 2977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:42.231577 2980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:42.232177 2969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:42.232786 2969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:42.233968 2969 hybrid_clock.cc:648] HybridClock initialized: now 1785984042233949 us; error 31 us; skew 500 ppm
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:42.236734 2969 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:42.237542 2969 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:42.237643 2969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:42.237874 2969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:42.239866 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "15dcccecb1584427b9a777dabe94f1e1"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "8abacf449a483f6be3e811c7b20fb122"
server_key_iv: "5f1a453c4ef3fa446b2c16a3176315fd"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.240442 2969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "15dcccecb1584427b9a777dabe94f1e1"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "8abacf449a483f6be3e811c7b20fb122"
server_key_iv: "5f1a453c4ef3fa446b2c16a3176315fd"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.244014 2969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:42.246407 2988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.247575 2969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:42.247717 2969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "15dcccecb1584427b9a777dabe94f1e1"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "8abacf449a483f6be3e811c7b20fb122"
server_key_iv: "5f1a453c4ef3fa446b2c16a3176315fd"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.247822 2969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:42.263837 2969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:42.267341 2969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:42.267552 2969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:42.268153 2969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:42.268996 2969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:42.269106 2969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.269200 2969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:42.269254 2969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.278493 2969 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:41845
I20260806 02:40:42.278525 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:41845 every 8 connection(s)
I20260806 02:40:42.279109 2969 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:42.285908 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 2969
I20260806 02:40:42.286046 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:42.286378 29436 external_mini_cluster.cc:1428] Setting key a090e56eb0621541c9c23bed98259b08
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:42.293915 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.280857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:47267 (local address 127.28.191.62:33367)
0806 02:40:42.280988 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.280995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.282126 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 02:40:42.283698 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.283701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.283762 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.283842 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:42.285261 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.286121 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.287548 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.287770 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.287891 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.290407 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:42.290410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:42.290412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:42.290436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:42.291821 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.292349 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.292354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.292361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.292438 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.292801 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.292804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.292806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.292989 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:42.293140 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.293616 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.293745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 02:40:42.294821 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.281386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33367 (local address 127.28.191.1:47267)
0806 02:40:42.281891 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.281954 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.283462 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.284223 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.284231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.284577 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:42.285106 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.285130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.286259 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.286263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.287422 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.287432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.287947 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.287951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.288104 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.288731 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:42.288748 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:42.290238 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:42.291943 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.291949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.291963 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.292236 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.292541 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.292544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.292546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.292693 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.293111 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:42.293118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:42.293373 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.293558 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.293875 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":150,"threads_started":1}
I20260806 02:40:42.296203 3157 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33367
I20260806 02:40:42.296425 3157 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:42.296880 3157 heartbeater.cc:511] Master 127.28.191.62:33367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:42.298413 2877 ts_manager.cc:194] Registered new tserver with Master: 15dcccecb1584427b9a777dabe94f1e1 (127.28.191.1:41845)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:40:42.299760 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:47267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:42.313803 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:42.426846 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:42.427153 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:42.427238 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:42.431094 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:42.431166 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:42.431249 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:42.436100 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:42.437052 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:42.438091 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:42.447376 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:42.447395 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:42.447606 3169 server_base.cc:1034] running on GCE node
W20260806 02:40:42.447376 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:42.448132 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:42.448747 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:42.449932 3169 hybrid_clock.cc:648] HybridClock initialized: now 1785984042449915 us; error 29 us; skew 500 ppm
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:42.452917 3169 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:42.453722 3169 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:42.453814 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:42.454036 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:42.455965 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "6386cafce9f043c59f01ddaf2acbaadb"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "4632cc17e8ce66b93df80114565c4e94"
server_key_iv: "678e6ae9111bfc05619351b7a946210e"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.456508 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "6386cafce9f043c59f01ddaf2acbaadb"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "4632cc17e8ce66b93df80114565c4e94"
server_key_iv: "678e6ae9111bfc05619351b7a946210e"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.460140 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:42.462477 3186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.463653 3169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:42.463789 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "6386cafce9f043c59f01ddaf2acbaadb"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "4632cc17e8ce66b93df80114565c4e94"
server_key_iv: "678e6ae9111bfc05619351b7a946210e"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.463894 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:42.477789 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:42.480839 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:42.481019 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:42.481563 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:42.482391 3169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:42.482438 3169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.482503 3169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:42.482576 3169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.491915 3169 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:42913
I20260806 02:40:42.491935 3349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:42913 every 8 connection(s)
I20260806 02:40:42.492462 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:42.500260 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 3169
I20260806 02:40:42.500391 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:42.500684 29436 external_mini_cluster.cc:1428] Setting key 6c18e63dc2e44c9317d22b3e7c7664be
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:42.505688 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.494245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57769 (local address 127.28.191.62:33367)
0806 02:40:42.494375 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.494378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.495178 (+ 800us) server_negotiation.cc:407] Connection header received
0806 02:40:42.496335 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.496337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.496380 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.496452 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:42.497657 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.498211 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.499299 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.499510 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.499594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.502099 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:42.502105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:42.502120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:42.502143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:42.503609 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.504128 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.504132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.504134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.504220 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.504592 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.504595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.504597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.504772 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:42.504879 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.505412 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.505531 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:40:42.506707 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.494599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33367 (local address 127.28.191.2:57769)
0806 02:40:42.495023 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.495066 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.496149 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.496614 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.496620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.496974 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:42.497511 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.497525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.498350 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.498353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.499171 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.499180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.499641 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.499643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.499774 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.500369 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:42.500396 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:42.501935 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:42.503734 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.503740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.503755 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.504018 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.504323 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.504326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.504328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.504480 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.504879 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:42.504885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:42.505180 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.505365 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.505681 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":126,"threads_started":1}
I20260806 02:40:42.507862 3350 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33367
I20260806 02:40:42.508111 3350 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:42.508510 3350 heartbeater.cc:511] Master 127.28.191.62:33367 requested a full tablet report, sending...
I20260806 02:40:42.509495 2877 ts_manager.cc:194] Registered new tserver with Master: 6386cafce9f043c59f01ddaf2acbaadb (127.28.191.2:42913)
I20260806 02:40:42.510013 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:57769
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:42.528842 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:42.642724 3360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:42.642983 3360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:42.643030 3360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:42.647158 3360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:42.647217 3360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:42.647286 3360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:42.652320 3360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:42.653224 3360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:42.654140 3360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:42.663262 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:42.663290 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:42.663304 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:42.663462 3360 server_base.cc:1034] running on GCE node
I20260806 02:40:42.663987 3360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:42.664618 3360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:42.665823 3360 hybrid_clock.cc:648] HybridClock initialized: now 1785984042665794 us; error 39 us; skew 500 ppm
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:42.668613 3360 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:42.669410 3360 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:42.669499 3360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:42.669740 3360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:42.671682 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "d8ba91b1e804445192265d3a502475c8"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "89dbcbe041d9d03a0ee815f059b4bca3"
server_key_iv: "577779a529ca8ab37764bb669e9531b5"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.672263 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "d8ba91b1e804445192265d3a502475c8"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "89dbcbe041d9d03a0ee815f059b4bca3"
server_key_iv: "577779a529ca8ab37764bb669e9531b5"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.675695 3360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:40:42.678035 3377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.679117 3360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:42.679260 3360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "d8ba91b1e804445192265d3a502475c8"
format_stamp: "Formatted at 2026-08-06 02:40:42 on dist-test-slave-tfpf"
server_key: "89dbcbe041d9d03a0ee815f059b4bca3"
server_key_iv: "577779a529ca8ab37764bb669e9531b5"
server_key_version: "encryptionkey@0"
I20260806 02:40:42.679363 3360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:42.705154 3360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:42.707966 3360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:42.708186 3360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:42.708750 3360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:42.709586 3360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:42.709658 3360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.709733 3360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:42.709785 3360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:42.718353 3360 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:34517
I20260806 02:40:42.718379 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:34517 every 8 connection(s)
I20260806 02:40:42.718946 3360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:42.725432 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 3360
I20260806 02:40:42.725593 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:42.725883 29436 external_mini_cluster.cc:1428] Setting key a3f1e1ca6bf3fa1024c23fda739e9689
Aug 06 02:40:42 dist-test-slave-tfpf krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984042, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:42.732452 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.720604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:44055 (local address 127.28.191.62:33367)
0806 02:40:42.720742 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.720746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.721533 (+ 787us) server_negotiation.cc:407] Connection header received
0806 02:40:42.722873 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.722876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.722928 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.723039 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:42.724201 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.724748 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.726069 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.726366 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.726516 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.729339 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:42.729341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:42.729343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:42.729366 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:42.730745 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.731195 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.731197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.731199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.731271 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.731563 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.731565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.731566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.731721 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:42.731789 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.732231 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.732323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:42.733319 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.720933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:33367 (local address 127.28.191.3:44055)
0806 02:40:42.721364 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.721400 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.722650 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.723194 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.723200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.723522 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:42.724064 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.724080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.724857 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.724859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.725931 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.725941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.726672 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.726676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.726861 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.727398 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:42.727413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:42.729194 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:42.730847 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.730851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.730861 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.731103 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.731342 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:42.731344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:42.731345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:42.731469 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:42.731801 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:42.731805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:42.732010 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.732156 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.732412 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260806 02:40:42.734421 3574 heartbeater.cc:347] Connected to a master server at 127.28.191.62:33367
I20260806 02:40:42.734652 3574 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:42.735090 3574 heartbeater.cc:511] Master 127.28.191.62:33367 requested a full tablet report, sending...
I20260806 02:40:42.736047 2877 ts_manager.cc:194] Registered new tserver with Master: d8ba91b1e804445192265d3a502475c8 (127.28.191.3:34517)
I20260806 02:40:42.736584 2877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:44055
I20260806 02:40:42.740034 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:40:42.749224 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.28.191.62:33367)
0806 02:40:42.742141 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.742146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.742316 (+ 170us) server_negotiation.cc:407] Connection header received
0806 02:40:42.742399 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.742401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.742437 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.742499 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:42.743191 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.743711 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.744452 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.744652 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.744725 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.746437 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:42.746440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:42.746442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:42.746462 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:42.747669 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.748075 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.748078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.748079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.748148 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:42.748374 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:42.748376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:42.748377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:42.748577 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:42.748695 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.748896 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.748982 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:40:42.751754 2877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:42.754856 2877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:42.769503 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.764339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.28.191.3:34517)
0806 02:40:42.764726 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.764738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.764873 (+ 135us) server_negotiation.cc:407] Connection header received
0806 02:40:42.764970 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.764974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.765176 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.765319 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:42.765676 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766411 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.767974 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.768721 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.768840 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.768998 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.769244 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.769340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":172,"threads_started":1}
I20260806 02:40:42.769488 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.763942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:34517 (local address 127.0.0.1:57706)
0806 02:40:42.764770 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.764783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.764876 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.765392 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.765395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.765414 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:42.765522 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.765530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766660 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.766665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.767848 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.767861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.768866 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.768870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.769006 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.769051 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.769200 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.769318 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":49,"thread_start_us":90,"threads_started":1}
I20260806 02:40:42.769825 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.764042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.28.191.1:41845)
0806 02:40:42.764868 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.764874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.764894 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:40:42.764968 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.764973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.765176 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.765319 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:42.765723 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766848 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.768321 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.768902 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.769006 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.769178 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.769555 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.769656 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":88,"threads_started":1}
I20260806 02:40:42.769876 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.763211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42913 (local address 127.0.0.1:38922)
0806 02:40:42.763597 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.763653 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.763777 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.764875 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.764882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.764904 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:42.765037 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.765049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766551 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.766554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.768147 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.768158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.769445 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.769449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.769569 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.769597 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.769685 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.769756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":76,"thread_start_us":101,"threads_started":1}
I20260806 02:40:42.770116 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.763567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:41845 (local address 127.0.0.1:35590)
0806 02:40:42.764489 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.764521 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.764632 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.765473 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.765476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.765493 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:42.765580 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.765584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766994 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.766998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.768227 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.768238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.769237 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.769241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.769322 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.769342 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.769411 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.769461 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":78,"thread_start_us":94,"threads_started":1}
I20260806 02:40:42.770295 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.763406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.28.191.2:42913)
0806 02:40:42.764180 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.764193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.764220 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:40:42.764298 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.764303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.764525 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.764677 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:42.765230 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.766405 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.768276 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.769128 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.769257 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.769377 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.770036 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.770130 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":102,"threads_started":1}
I20260806 02:40:42.773180 3253 tablet_service.cc:1467] Processing CreateTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be (DEFAULT_TABLE table=test-table [id=14e396bca371417296b613e3a011d559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:42.773144 3460 tablet_service.cc:1467] Processing CreateTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be (DEFAULT_TABLE table=test-table [id=14e396bca371417296b613e3a011d559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:42.773139 3061 tablet_service.cc:1467] Processing CreateTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be (DEFAULT_TABLE table=test-table [id=14e396bca371417296b613e3a011d559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:42.774138 3253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec609fdf243c4e26a6ac923b21b1d3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:42.774163 3061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec609fdf243c4e26a6ac923b21b1d3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:42.774175 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec609fdf243c4e26a6ac923b21b1d3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:42.780210 3601 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Bootstrap starting.
I20260806 02:40:42.781865 3604 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: Bootstrap starting.
I20260806 02:40:42.782394 3601 tablet_bootstrap.cc:654] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:42.783591 3601 log.cc:824] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:42.783802 3604 tablet_bootstrap.cc:654] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:42.785107 3604 log.cc:824] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:42.785524 3602 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Bootstrap starting.
I20260806 02:40:42.785750 3601 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: No bootstrap required, opened a new log
I20260806 02:40:42.785933 3601 ts_tablet_manager.cc:1397] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 02:40:42.787250 3602 tablet_bootstrap.cc:654] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:42.788234 3602 log.cc:824] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:42.788710 3601 raft_consensus.cc:348] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.789001 3601 raft_consensus.cc:374] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:42.789098 3601 raft_consensus.cc:729] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6386cafce9f043c59f01ddaf2acbaadb, State: Initialized, Role: FOLLOWER
I20260806 02:40:42.789564 3601 consensus_queue.cc:260] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.790025 3602 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: No bootstrap required, opened a new log
I20260806 02:40:42.790213 3602 ts_tablet_manager.cc:1397] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:40:42.790203 3350 heartbeater.cc:503] Master 127.28.191.62:33367 was elected leader, sending a full tablet report...
I20260806 02:40:42.790467 3601 ts_tablet_manager.cc:1428] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260806 02:40:42.791653 3604 tablet_bootstrap.cc:492] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: No bootstrap required, opened a new log
I20260806 02:40:42.791869 3604 ts_tablet_manager.cc:1397] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 02:40:42.793912 3602 raft_consensus.cc:348] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.794236 3602 raft_consensus.cc:374] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:42.794322 3602 raft_consensus.cc:729] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ba91b1e804445192265d3a502475c8, State: Initialized, Role: FOLLOWER
I20260806 02:40:42.794754 3602 consensus_queue.cc:260] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.795166 3604 raft_consensus.cc:348] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.795411 3604 raft_consensus.cc:374] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:42.795475 3574 heartbeater.cc:503] Master 127.28.191.62:33367 was elected leader, sending a full tablet report...
I20260806 02:40:42.795485 3604 raft_consensus.cc:729] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dcccecb1584427b9a777dabe94f1e1, State: Initialized, Role: FOLLOWER
I20260806 02:40:42.795817 3602 ts_tablet_manager.cc:1428] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260806 02:40:42.795913 3604 consensus_queue.cc:260] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.796500 3157 heartbeater.cc:503] Master 127.28.191.62:33367 was elected leader, sending a full tablet report...
I20260806 02:40:42.796857 3604 ts_tablet_manager.cc:1428] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:40:42.908336 3612 raft_consensus.cc:482] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:42.908574 3612 raft_consensus.cc:504] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.909896 3612 leader_election.cc:288] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6386cafce9f043c59f01ddaf2acbaadb (127.28.191.2:42913), 15dcccecb1584427b9a777dabe94f1e1 (127.28.191.1:41845)
I20260806 02:40:42.914145 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.910280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:39595 (local address 127.28.191.2:42913)
0806 02:40:42.910395 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.910399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.910415 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:40:42.910587 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.910591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.910643 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.910732 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:42.911084 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.911689 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913164 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.913573 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913705 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.913761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.913949 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.914019 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 02:40:42.914171 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.910178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42913 (local address 127.28.191.3:39595)
0806 02:40:42.910329 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.910345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.910447 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.910751 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.910753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.910771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:42.910943 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.910949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.911794 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.911797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.913019 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.913030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.913697 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.913803 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.913824 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.913934 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.914003 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":60,"spinlock_wait_cycles":2816}
I20260806 02:40:42.914219 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.910178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:41845 (local address 127.28.191.3:46895)
0806 02:40:42.910712 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:42.910728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:42.910844 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:42.911118 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:42.911121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:42.911138 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:42.911244 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.911251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.912001 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.912005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.913102 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:42.913114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.913754 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:42.913861 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.913883 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:40:42.913988 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:40:42.914058 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":165,"spinlock_wait_cycles":23552,"thread_start_us":140,"threads_started":1}
I20260806 02:40:42.914207 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.910298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:46895 (local address 127.28.191.1:41845)
0806 02:40:42.910434 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.910441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.910817 (+ 376us) server_negotiation.cc:407] Connection header received
0806 02:40:42.910960 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.910963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.911020 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.911095 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:42.911359 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.911881 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913213 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.913629 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.913762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.913816 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.914024 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.914105 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:40:42.914987 3281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec609fdf243c4e26a6ac923b21b1d3be" candidate_uuid: "d8ba91b1e804445192265d3a502475c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6386cafce9f043c59f01ddaf2acbaadb" is_pre_election: true
I20260806 02:40:42.915113 3094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec609fdf243c4e26a6ac923b21b1d3be" candidate_uuid: "d8ba91b1e804445192265d3a502475c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dcccecb1584427b9a777dabe94f1e1" is_pre_election: true
I20260806 02:40:42.915337 3281 raft_consensus.cc:2454] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ba91b1e804445192265d3a502475c8 in term 0.
I20260806 02:40:42.915382 3094 raft_consensus.cc:2454] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8ba91b1e804445192265d3a502475c8 in term 0.
I20260806 02:40:42.915880 3379 leader_election.cc:302] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6386cafce9f043c59f01ddaf2acbaadb, d8ba91b1e804445192265d3a502475c8; no voters:
I20260806 02:40:42.916146 3612 raft_consensus.cc:2781] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:42.916236 3612 raft_consensus.cc:482] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:42.916289 3612 raft_consensus.cc:3037] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:42.917361 3612 raft_consensus.cc:504] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.917701 3612 leader_election.cc:288] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [CANDIDATE]: Term 1 election: Requested vote from peers 6386cafce9f043c59f01ddaf2acbaadb (127.28.191.2:42913), 15dcccecb1584427b9a777dabe94f1e1 (127.28.191.1:41845)
I20260806 02:40:42.918097 3281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec609fdf243c4e26a6ac923b21b1d3be" candidate_uuid: "d8ba91b1e804445192265d3a502475c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6386cafce9f043c59f01ddaf2acbaadb"
I20260806 02:40:42.918144 3094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec609fdf243c4e26a6ac923b21b1d3be" candidate_uuid: "d8ba91b1e804445192265d3a502475c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dcccecb1584427b9a777dabe94f1e1"
I20260806 02:40:42.918234 3281 raft_consensus.cc:3037] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:42.918264 3094 raft_consensus.cc:3037] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:42.919525 3094 raft_consensus.cc:2454] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ba91b1e804445192265d3a502475c8 in term 1.
I20260806 02:40:42.919525 3281 raft_consensus.cc:2454] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8ba91b1e804445192265d3a502475c8 in term 1.
I20260806 02:40:42.919844 3379 leader_election.cc:302] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6386cafce9f043c59f01ddaf2acbaadb, d8ba91b1e804445192265d3a502475c8; no voters:
I20260806 02:40:42.920009 3612 raft_consensus.cc:2781] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:42.920262 3612 raft_consensus.cc:686] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 1 LEADER]: Becoming Leader. State: Replica: d8ba91b1e804445192265d3a502475c8, State: Running, Role: LEADER
I20260806 02:40:42.920626 3612 consensus_queue.cc:237] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } }
I20260806 02:40:42.924026 2877 catalog_manager.cc:5495] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8ba91b1e804445192265d3a502475c8 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "d8ba91b1e804445192265d3a502475c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8ba91b1e804445192265d3a502475c8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:42.957751 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:42.954292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.28.191.3:34517)
0806 02:40:42.954422 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:42.954425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:42.954439 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:42.954510 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:42.954512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:42.954561 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:42.954644 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:42.954878 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.955364 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.956290 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:42.956497 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:42.956575 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:42.956848 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:42.957369 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:42.957459 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:42.957521 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:40:42.957569 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:40:42.963483 3281 raft_consensus.cc:1261] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 1 FOLLOWER]: Refusing update from remote peer d8ba91b1e804445192265d3a502475c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:42.963483 3094 raft_consensus.cc:1261] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 1 FOLLOWER]: Refusing update from remote peer d8ba91b1e804445192265d3a502475c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:42.964187 3612 consensus_queue.cc:1035] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6386cafce9f043c59f01ddaf2acbaadb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:42.964423 3615 consensus_queue.cc:1035] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15dcccecb1584427b9a777dabe94f1e1" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:40:42.970296 3575 tablet.cc:2369] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:42.973482 3618 mvcc.cc:205] Tried to move back new op lower bound from 7315390639971442688 to 7315390639806201856. Current Snapshot: MvccSnapshot[applied={T|T < 7315390639971442688}]
I20260806 02:40:42.975344 3621 mvcc.cc:205] Tried to move back new op lower bound from 7315390639971442688 to 7315390639806201856. Current Snapshot: MvccSnapshot[applied={T|T < 7315390639971442688}]
I20260806 02:40:42.975685 3619 mvcc.cc:205] Tried to move back new op lower bound from 7315390639971442688 to 7315390639806201856. Current Snapshot: MvccSnapshot[applied={T|T < 7315390639971442688}]
I20260806 02:40:42.983680 2876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:42.983814 2876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:42.986338 2876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb8b8376db454e779231ff3feba662bb: Sending DeleteTablet for 3 replicas of tablet ec609fdf243c4e26a6ac923b21b1d3be
I20260806 02:40:42.987128 3460 tablet_service.cc:1514] Processing DeleteTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:42 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57706
I20260806 02:40:42.987299 3061 tablet_service.cc:1514] Processing DeleteTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:42 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:35590
I20260806 02:40:42.987303 3253 tablet_service.cc:1514] Processing DeleteTablet for tablet ec609fdf243c4e26a6ac923b21b1d3be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:42 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:38922
I20260806 02:40:42.987759 3643 tablet_replica.cc:330] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1: stopping tablet replica
I20260806 02:40:42.987845 3642 tablet_replica.cc:330] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: stopping tablet replica
I20260806 02:40:42.988125 3643 raft_consensus.cc:2229] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:42.988287 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2969
I20260806 02:40:42.988430 3643 raft_consensus.cc:2258] T ec609fdf243c4e26a6ac923b21b1d3be P 15dcccecb1584427b9a777dabe94f1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:42.998899 3644 tablet_replica.cc:330] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: stopping tablet replica
I20260806 02:40:42.999329 3644 raft_consensus.cc:2229] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:42.999704 3644 raft_consensus.cc:2258] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:43.002354 3644 ts_tablet_manager.cc:1905] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:43.005299 3644 ts_tablet_manager.cc:1918] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:43.005398 3644 log.cc:1196] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/ec609fdf243c4e26a6ac923b21b1d3be
I20260806 02:40:43.005705 3644 ts_tablet_manager.cc:1939] T ec609fdf243c4e26a6ac923b21b1d3be P 6386cafce9f043c59f01ddaf2acbaadb: Deleting consensus metadata
I20260806 02:40:43.007136 2856 catalog_manager.cc:4841] TS 6386cafce9f043c59f01ddaf2acbaadb (127.28.191.2:42913): tablet ec609fdf243c4e26a6ac923b21b1d3be (table test-table [id=14e396bca371417296b613e3a011d559]) successfully deleted
I20260806 02:40:42.988180 3642 raft_consensus.cc:2229] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:43.011384 3642 raft_consensus.cc:2258] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:43.013106 3642 ts_tablet_manager.cc:1905] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:43.015846 3642 ts_tablet_manager.cc:1918] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:43.016002 3642 log.cc:1196] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/ec609fdf243c4e26a6ac923b21b1d3be
I20260806 02:40:43.016390 3642 ts_tablet_manager.cc:1939] T ec609fdf243c4e26a6ac923b21b1d3be P d8ba91b1e804445192265d3a502475c8: Deleting consensus metadata
I20260806 02:40:43.017669 2856 catalog_manager.cc:4841] TS d8ba91b1e804445192265d3a502475c8 (127.28.191.3:34517): tablet ec609fdf243c4e26a6ac923b21b1d3be (table test-table [id=14e396bca371417296b613e3a011d559]) successfully deleted
W20260806 02:40:43.020011 2858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:41845 (error 108)
I20260806 02:40:43.021031 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.020741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:41845 (local address 127.0.0.1:35592)
0806 02:40:43.020886 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.020952 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:41845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:40:43.021188 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 3169
W20260806 02:40:43.021226 2858 catalog_manager.cc:4584] TS 15dcccecb1584427b9a777dabe94f1e1 (127.28.191.1:41845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec609fdf243c4e26a6ac923b21b1d3be: Network error: Client connection negotiation failed: client connection to 127.28.191.1:41845: connect: Connection refused (error 111)
I20260806 02:40:43.030747 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 3360
I20260806 02:40:43.039580 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 2833
2026-08-06T02:40:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2545 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:43Z Disabled control of system clock
I20260806 02:40:43.075979 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36137
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:40:43.184911 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:43.185187 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:43.185226 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:43.189666 3651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:43.189723 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:43.189740 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:43.189756 3651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:43.189771 3651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:43.194491 3651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:43.195489 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:43.196822 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:43.205148 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.205257 3651 server_base.cc:1034] running on GCE node
W20260806 02:40:43.205150 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.205194 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.205924 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:43.206893 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:43.208052 3651 hybrid_clock.cc:648] HybridClock initialized: now 1785984043208022 us; error 41 us; skew 500 ppm
I20260806 02:40:43.209901 3651 webserver.cc:460] Webserver started at http://127.28.191.62:40703/ using document root <none> and password file <none>
I20260806 02:40:43.210429 3651 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:43.210489 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:43.210704 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:43.212769 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "0879f772940742d0aaf0aab6a6555a08"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "12d4d21a457489eca5ebe062949142c9"
server_key_iv: "71f29865501526e4d8f64494b7d55619"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.213361 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "0879f772940742d0aaf0aab6a6555a08"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "12d4d21a457489eca5ebe062949142c9"
server_key_iv: "71f29865501526e4d8f64494b7d55619"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.217051 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:40:43.219363 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.220515 3651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:40:43.220659 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "0879f772940742d0aaf0aab6a6555a08"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "12d4d21a457489eca5ebe062949142c9"
server_key_iv: "71f29865501526e4d8f64494b7d55619"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.220772 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:43.225312 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:43.225901 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:43.226090 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:43.233296 3651 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36137
I20260806 02:40:43.233330 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36137 every 8 connection(s)
I20260806 02:40:43.234201 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:43.237463 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:43.241751 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 3651
I20260806 02:40:43.241883 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:43.242144 29436 external_mini_cluster.cc:1428] Setting key 38fef8306f5ea3c68fc1ca48bebb68e3
I20260806 02:40:43.243083 3754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: Bootstrap starting.
I20260806 02:40:43.245235 3754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:43.246147 3754 log.cc:824] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:43.247668 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.243574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.28.191.62:36137)
0806 02:40:43.244170 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.244185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.244220 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:40:43.244834 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.244850 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.245252 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.245564 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.246436 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.246441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.246451 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.246462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.246544 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.246571 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.246704 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.246795 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.247033 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260806 02:40:43.248144 3754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: No bootstrap required, opened a new log
I20260806 02:40:43.251838 3754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } }
I20260806 02:40:43.252159 3754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:43.252233 3754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0879f772940742d0aaf0aab6a6555a08, State: Initialized, Role: FOLLOWER
I20260806 02:40:43.252723 3754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } }
I20260806 02:40:43.252918 3754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:43.252995 3754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:43.253126 3754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:43.253933 3754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } }
I20260806 02:40:43.254269 3754 leader_election.cc:302] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0879f772940742d0aaf0aab6a6555a08; no voters:
I20260806 02:40:43.254503 3754 leader_election.cc:288] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:43.254627 3760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:43.254875 3760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [term 1 LEADER]: Becoming Leader. State: Replica: 0879f772940742d0aaf0aab6a6555a08, State: Running, Role: LEADER
I20260806 02:40:43.255179 3760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } }
I20260806 02:40:43.255826 3754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:43.257675 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0879f772940742d0aaf0aab6a6555a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } } }
I20260806 02:40:43.257799 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:43.258064 3764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0879f772940742d0aaf0aab6a6555a08. Latest consensus state: current_term: 1 leader_uuid: "0879f772940742d0aaf0aab6a6555a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0879f772940742d0aaf0aab6a6555a08" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36137 } } }
I20260806 02:40:43.258188 3764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:43.258232 3777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:43.261346 3777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:43.266790 3777 catalog_manager.cc:1347] Generated new cluster ID: 76e5e822e668483bb6d46f790056ae47
I20260806 02:40:43.266867 3777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:43.277854 3777 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:43.278194 3777 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:43.285341 3777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: Generated new TSK 0
I20260806 02:40:43.286036 3777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:40:43.288298 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--builtin_ntp_servers=127.28.191.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:40:43.400234 3786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:43.400454 3786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:43.400490 3786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:43.404697 3786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:43.404743 3786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:43.404825 3786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:43.409231 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:43.410176 3786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:43.411371 3786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:43.420511 3795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.420512 3796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.420746 3786 server_base.cc:1034] running on GCE node
W20260806 02:40:43.420504 3798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.421260 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:43.421862 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:43.423048 3786 hybrid_clock.cc:648] HybridClock initialized: now 1785984043423043 us; error 32 us; skew 500 ppm
I20260806 02:40:43.424984 3786 webserver.cc:460] Webserver started at http://127.28.191.1:36169/ using document root <none> and password file <none>
I20260806 02:40:43.425566 3786 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:43.425668 3786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:43.425901 3786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:43.427919 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "9449c423c0534feb98768ffdd9d9b6d4"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "18e1e2eefab9224e7c2f191bfe48770b"
server_key_iv: "c0dbfbd1233bbc63dc18e05c87227c9e"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.428565 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "9449c423c0534feb98768ffdd9d9b6d4"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "18e1e2eefab9224e7c2f191bfe48770b"
server_key_iv: "c0dbfbd1233bbc63dc18e05c87227c9e"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.432169 3786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:43.434728 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.435877 3786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 02:40:43.436026 3786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "9449c423c0534feb98768ffdd9d9b6d4"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "18e1e2eefab9224e7c2f191bfe48770b"
server_key_iv: "c0dbfbd1233bbc63dc18e05c87227c9e"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.436137 3786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:43.440670 3786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:43.441290 3786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:43.441468 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:43.441979 3786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:43.442869 3786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:43.442941 3786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.443005 3786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:43.443054 3786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.451413 3786 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42975
I20260806 02:40:43.451432 3970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42975 every 8 connection(s)
I20260806 02:40:43.452231 3786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:43.455452 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 3786
I20260806 02:40:43.455603 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:43.455899 29436 external_mini_cluster.cc:1428] Setting key 32cbc8c4d093086456053331d4625d21
I20260806 02:40:43.458022 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--builtin_ntp_servers=127.28.191.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:40:43.458657 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.454092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:46329 (local address 127.28.191.62:36137)
0806 02:40:43.454218 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.454222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.455181 (+ 959us) server_negotiation.cc:407] Connection header received
0806 02:40:43.456878 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.456881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.456923 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.457002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.458221 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.458224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.458226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.458241 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.458262 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.458269 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.458313 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.458488 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.458529 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:40:43.459415 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.454437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36137 (local address 127.28.191.1:46329)
0806 02:40:43.455017 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.455063 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.456674 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.457209 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.457217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.457321 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.457951 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.457952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.457987 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.458055 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.458367 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.458373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.458376 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.458527 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.458738 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260806 02:40:43.460723 3972 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36137
I20260806 02:40:43.460938 3972 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:43.461376 3972 heartbeater.cc:511] Master 127.28.191.62:36137 requested a full tablet report, sending...
I20260806 02:40:43.463045 3701 ts_manager.cc:194] Registered new tserver with Master: 9449c423c0534feb98768ffdd9d9b6d4 (127.28.191.1:42975)
W20260806 02:40:43.570958 3978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:43.571213 3978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:43.571250 3978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:43.575184 3978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:43.575242 3978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:43.575313 3978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:43.579856 3978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:43.580869 3978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:43.582289 3978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:43.592243 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.592243 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.592271 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.592461 3978 server_base.cc:1034] running on GCE node
I20260806 02:40:43.592895 3978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:43.593597 3978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:43.594791 3978 hybrid_clock.cc:648] HybridClock initialized: now 1785984043594763 us; error 46 us; skew 500 ppm
I20260806 02:40:43.596578 3978 webserver.cc:460] Webserver started at http://127.28.191.2:45717/ using document root <none> and password file <none>
I20260806 02:40:43.597229 3978 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:43.597326 3978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:43.597573 3978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:43.599622 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "87d4bb12c08746619c0e2d8417c24794"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "15d91879e8d86bf930f57a67a9ea8330"
server_key_iv: "3d4d8cdcdf922d6fe5f9dbcc76909739"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.600194 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "87d4bb12c08746619c0e2d8417c24794"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "15d91879e8d86bf930f57a67a9ea8330"
server_key_iv: "3d4d8cdcdf922d6fe5f9dbcc76909739"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.603955 3978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:43.606705 3998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.607920 3978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:43.608058 3978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "87d4bb12c08746619c0e2d8417c24794"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "15d91879e8d86bf930f57a67a9ea8330"
server_key_iv: "3d4d8cdcdf922d6fe5f9dbcc76909739"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.608202 3978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:43.612740 3978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:43.613417 3978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:43.613596 3978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:43.614132 3978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:43.614987 3978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:43.615067 3978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.615159 3978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:43.615209 3978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.623993 3978 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:34345
I20260806 02:40:43.624011 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:34345 every 8 connection(s)
I20260806 02:40:43.624897 3978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:43.626398 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 3978
I20260806 02:40:43.626538 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:43.626865 29436 external_mini_cluster.cc:1428] Setting key 3ff33253c2f241d31adf504d83c0a91a
I20260806 02:40:43.628994 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--builtin_ntp_servers=127.28.191.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:40:43.631572 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.626752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:60297 (local address 127.28.191.62:36137)
0806 02:40:43.626885 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.626889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.628234 (+ 1345us) server_negotiation.cc:407] Connection header received
0806 02:40:43.629725 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.629731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.629778 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.629869 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.631141 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.631144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.631146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.631157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.631177 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.631185 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.631257 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.631431 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.631466 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:40:43.632718 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.627276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36137 (local address 127.28.191.2:60297)
0806 02:40:43.628035 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.628117 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.629520 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.630033 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.630051 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.630130 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.630788 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.630792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.630853 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.630956 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.631274 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.631286 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.631294 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.631474 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.631724 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":138,"threads_started":1}
I20260806 02:40:43.633868 4183 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36137
I20260806 02:40:43.634140 4183 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:43.634626 4183 heartbeater.cc:511] Master 127.28.191.62:36137 requested a full tablet report, sending...
I20260806 02:40:43.635741 3701 ts_manager.cc:194] Registered new tserver with Master: 87d4bb12c08746619c0e2d8417c24794 (127.28.191.2:34345)
W20260806 02:40:43.742463 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:43.742684 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:43.742741 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:43.746767 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:43.746814 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:43.746905 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:43.751359 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:43.752271 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:43.753625 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:43.763031 4193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.763105 4197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:43.763031 4195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:43.763201 4186 server_base.cc:1034] running on GCE node
I20260806 02:40:43.763758 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:43.764397 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:43.765578 4186 hybrid_clock.cc:648] HybridClock initialized: now 1785984043765561 us; error 31 us; skew 500 ppm
I20260806 02:40:43.767400 4186 webserver.cc:460] Webserver started at http://127.28.191.3:33223/ using document root <none> and password file <none>
I20260806 02:40:43.767942 4186 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:43.768039 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:43.768263 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:43.770283 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "213b19012ec04aaf88b9cb4882867e2b"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "2b4d04fb0b0ab546960893c49af7feae"
server_key_iv: "3a390f5946d56e50f04b926aba571572"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.770850 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "213b19012ec04aaf88b9cb4882867e2b"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "2b4d04fb0b0ab546960893c49af7feae"
server_key_iv: "3a390f5946d56e50f04b926aba571572"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.774399 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:43.776705 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.777940 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:43.778095 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "213b19012ec04aaf88b9cb4882867e2b"
format_stamp: "Formatted at 2026-08-06 02:40:43 on dist-test-slave-tfpf"
server_key: "2b4d04fb0b0ab546960893c49af7feae"
server_key_iv: "3a390f5946d56e50f04b926aba571572"
server_key_version: "encryptionkey@0"
I20260806 02:40:43.778216 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:43.782577 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:43.783182 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:43.783360 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:43.783871 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:43.784790 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:43.784874 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.784936 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:43.784986 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:43.793210 4186 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36653
I20260806 02:40:43.793244 4375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36653 every 8 connection(s)
I20260806 02:40:43.794067 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:43.796927 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 4186
I20260806 02:40:43.797096 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:43.797412 29436 external_mini_cluster.cc:1428] Setting key 01672ed121209f6cbc22b9eeb0ddd484
I20260806 02:40:43.800401 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.795812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:56781 (local address 127.28.191.62:36137)
0806 02:40:43.795952 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.795955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.797121 (+ 1166us) server_negotiation.cc:407] Connection header received
0806 02:40:43.798840 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.798844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.798887 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.798970 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.800040 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.800044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.800046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.800057 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.800076 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.800083 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.800154 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.800268 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.800303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:43.801283 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.796201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36137 (local address 127.28.191.3:56781)
0806 02:40:43.796943 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.796999 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.798595 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.799113 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.799119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.799199 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.799801 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.799803 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.799853 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.799914 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.800156 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.800162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.800166 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.800285 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.800452 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260806 02:40:43.802289 4377 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36137
I20260806 02:40:43.802508 4377 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:43.802917 4377 heartbeater.cc:511] Master 127.28.191.62:36137 requested a full tablet report, sending...
I20260806 02:40:43.803874 3701 ts_manager.cc:194] Registered new tserver with Master: 213b19012ec04aaf88b9cb4882867e2b (127.28.191.3:36653)
I20260806 02:40:43.811261 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:40:43.814617 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.813235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.28.191.62:36137)
0806 02:40:43.813355 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.813358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.813540 (+ 182us) server_negotiation.cc:407] Connection header received
0806 02:40:43.813629 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.813632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.813663 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.813729 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.814348 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.814350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.814352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.814363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.814379 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.814386 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.814455 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.814495 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.814525 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:40:43.817250 3701 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:43.820091 3701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:43.831321 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.827974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.28.191.3:36653)
0806 02:40:43.829028 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.829037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.829097 (+ 60us) server_negotiation.cc:407] Connection header received
0806 02:40:43.829152 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.829157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.829352 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.829497 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.830804 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.830809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.830818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.830830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.830870 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.830899 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.831025 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.831107 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.831182 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":154,"threads_started":1}
I20260806 02:40:43.831306 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.828462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42975 (local address 127.0.0.1:41674)
0806 02:40:43.828801 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.828812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.828871 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.829855 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.829858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.829874 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.830440 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.830442 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.830454 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.830493 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.830915 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.830924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.830930 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.831049 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.831121 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":801,"threads_started":1}
I20260806 02:40:43.831437 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.828656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.28.191.1:42975)
0806 02:40:43.829421 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.829450 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.829475 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:40:43.829524 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.829529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.829687 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.830062 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.830717 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.830726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.830740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.830752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.830793 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.830820 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.830931 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.831221 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.831287 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":144,"threads_started":1}
I20260806 02:40:43.831532 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.827816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:36653 (local address 127.0.0.1:37432)
0806 02:40:43.828322 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.828382 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.828495 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.829511 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.829517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.829589 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.830318 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.830320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.830343 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.830662 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.831010 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.831013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.831016 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.831097 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.831143 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":70,"threads_started":1}
I20260806 02:40:43.831671 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.829008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.28.191.2:34345)
0806 02:40:43.829857 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:43.829869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:43.829892 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:40:43.829942 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:43.829947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:43.830119 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:43.830254 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:43.830862 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:43.830888 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:43.830903 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:43.830913 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:43.830954 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:43.830982 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:43.831389 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:43.831434 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 02:40:43.831498 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":133,"threads_started":1}
I20260806 02:40:43.831311 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:43.828014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34345 (local address 127.0.0.1:55316)
0806 02:40:43.828718 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:43.828731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:43.828801 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:43.830237 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:43.830240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:43.830254 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:43.830670 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:43.830672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:43.830681 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:43.830703 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:43.831098 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:43.831100 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:43.831104 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:40:43.831154 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 02:40:43.831182 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":120,"threads_started":1}
I20260806 02:40:43.834338 4069 tablet_service.cc:1467] Processing CreateTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 (DEFAULT_TABLE table=test-table [id=dad3265563254416a1415cac928dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:43.834339 3880 tablet_service.cc:1467] Processing CreateTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 (DEFAULT_TABLE table=test-table [id=dad3265563254416a1415cac928dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:43.835386 4069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39f15d97e4b40aaba91245fc1cefff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:43.835387 3880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39f15d97e4b40aaba91245fc1cefff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:43.836683 4281 tablet_service.cc:1467] Processing CreateTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 (DEFAULT_TABLE table=test-table [id=dad3265563254416a1415cac928dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:43.837685 4281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39f15d97e4b40aaba91245fc1cefff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:43.841567 4404 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Bootstrap starting.
I20260806 02:40:43.842922 4406 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Bootstrap starting.
I20260806 02:40:43.844333 4404 tablet_bootstrap.cc:654] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:43.844578 4408 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Bootstrap starting.
I20260806 02:40:43.844933 4406 tablet_bootstrap.cc:654] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:43.845367 4404 log.cc:824] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:43.845840 4406 log.cc:824] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:43.846642 4408 tablet_bootstrap.cc:654] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:43.847260 4408 log.cc:824] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:43.847369 4404 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: No bootstrap required, opened a new log
I20260806 02:40:43.847599 4404 ts_tablet_manager.cc:1397] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:40:43.847725 4406 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: No bootstrap required, opened a new log
I20260806 02:40:43.847895 4406 ts_tablet_manager.cc:1397] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:40:43.848845 4408 tablet_bootstrap.cc:492] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: No bootstrap required, opened a new log
I20260806 02:40:43.849027 4408 ts_tablet_manager.cc:1397] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:40:43.852087 4404 raft_consensus.cc:348] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.852087 4406 raft_consensus.cc:348] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.852352 4406 raft_consensus.cc:374] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:43.852352 4404 raft_consensus.cc:374] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:43.852416 4404 raft_consensus.cc:729] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9449c423c0534feb98768ffdd9d9b6d4, State: Initialized, Role: FOLLOWER
I20260806 02:40:43.852432 4406 raft_consensus.cc:729] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d4bb12c08746619c0e2d8417c24794, State: Initialized, Role: FOLLOWER
I20260806 02:40:43.852288 4408 raft_consensus.cc:348] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.852552 4408 raft_consensus.cc:374] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:43.852641 4408 raft_consensus.cc:729] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b19012ec04aaf88b9cb4882867e2b, State: Initialized, Role: FOLLOWER
I20260806 02:40:43.852939 4406 consensus_queue.cc:260] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.852939 4404 consensus_queue.cc:260] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.853132 4408 consensus_queue.cc:260] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:43.853645 3972 heartbeater.cc:503] Master 127.28.191.62:36137 was elected leader, sending a full tablet report...
I20260806 02:40:43.853943 4406 ts_tablet_manager.cc:1428] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 02:40:43.853983 4408 ts_tablet_manager.cc:1428] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260806 02:40:43.854286 4377 heartbeater.cc:503] Master 127.28.191.62:36137 was elected leader, sending a full tablet report...
I20260806 02:40:43.854707 4183 heartbeater.cc:503] Master 127.28.191.62:36137 was elected leader, sending a full tablet report...
I20260806 02:40:43.855837 4404 ts_tablet_manager.cc:1428] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260806 02:40:43.875810 4184 tablet.cc:2369] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:40:43.953438 3973 tablet.cc:2369] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:40:44.045118 4379 tablet.cc:2369] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:44.057116 4418 raft_consensus.cc:482] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:44.057363 4418 raft_consensus.cc:504] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:44.058657 4418 leader_election.cc:288] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87d4bb12c08746619c0e2d8417c24794 (127.28.191.2:34345), 9449c423c0534feb98768ffdd9d9b6d4 (127.28.191.1:42975)
I20260806 02:40:44.060708 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.058900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42975 (local address 127.28.191.3:58883)
0806 02:40:44.059085 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:44.059101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:44.059179 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:44.059458 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:44.059463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:44.059484 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:44.060058 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:44.060060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:44.060071 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:44.060098 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:44.060413 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:44.060416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:44.060419 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:40:44.060538 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:40:44.060575 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:40:44.060730 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.059102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:58883 (local address 127.28.191.1:42975)
0806 02:40:44.059228 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.059233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.059247 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:44.059298 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.059303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.059349 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.059438 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:44.060250 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:44.060256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:44.060260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:44.060276 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:44.060300 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:44.060311 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:44.060415 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:44.060536 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:40:44.060580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:40:44.061139 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.059152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:57407 (local address 127.28.191.2:34345)
0806 02:40:44.059282 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.059286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.059734 (+ 448us) server_negotiation.cc:407] Connection header received
0806 02:40:44.059846 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.059852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.059895 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.059974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:44.060679 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:44.060682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:44.060685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:44.060696 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:44.060716 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:44.060725 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:44.060818 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:44.060952 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 02:40:44.060989 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:40:44.061153 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.058900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34345 (local address 127.28.191.3:57407)
0806 02:40:44.059647 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:44.059663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:44.059756 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:44.060031 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:44.060036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:44.060053 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:44.060511 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:40:44.060515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:44.060526 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:40:44.060550 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:44.060863 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:44.060866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:44.060870 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:40:44.060946 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:40:44.060979 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":302,"thread_start_us":137,"threads_started":1}
I20260806 02:40:44.061494 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39f15d97e4b40aaba91245fc1cefff0" candidate_uuid: "213b19012ec04aaf88b9cb4882867e2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9449c423c0534feb98768ffdd9d9b6d4" is_pre_election: true
I20260806 02:40:44.061786 3908 raft_consensus.cc:2454] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b19012ec04aaf88b9cb4882867e2b in term 0.
I20260806 02:40:44.061802 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39f15d97e4b40aaba91245fc1cefff0" candidate_uuid: "213b19012ec04aaf88b9cb4882867e2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d4bb12c08746619c0e2d8417c24794" is_pre_election: true
I20260806 02:40:44.062000 4104 raft_consensus.cc:2454] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b19012ec04aaf88b9cb4882867e2b in term 0.
I20260806 02:40:44.062366 4214 leader_election.cc:302] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b19012ec04aaf88b9cb4882867e2b, 9449c423c0534feb98768ffdd9d9b6d4; no voters:
I20260806 02:40:44.062635 4418 raft_consensus.cc:2781] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:44.062718 4418 raft_consensus.cc:482] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:44.062762 4418 raft_consensus.cc:3037] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:44.063798 4418 raft_consensus.cc:504] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:44.064113 4418 leader_election.cc:288] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [CANDIDATE]: Term 1 election: Requested vote from peers 87d4bb12c08746619c0e2d8417c24794 (127.28.191.2:34345), 9449c423c0534feb98768ffdd9d9b6d4 (127.28.191.1:42975)
I20260806 02:40:44.064406 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39f15d97e4b40aaba91245fc1cefff0" candidate_uuid: "213b19012ec04aaf88b9cb4882867e2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d4bb12c08746619c0e2d8417c24794"
I20260806 02:40:44.064460 3908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39f15d97e4b40aaba91245fc1cefff0" candidate_uuid: "213b19012ec04aaf88b9cb4882867e2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9449c423c0534feb98768ffdd9d9b6d4"
I20260806 02:40:44.064543 4104 raft_consensus.cc:3037] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:44.064631 3908 raft_consensus.cc:3037] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:44.065646 4104 raft_consensus.cc:2454] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b19012ec04aaf88b9cb4882867e2b in term 1.
I20260806 02:40:44.065842 3908 raft_consensus.cc:2454] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b19012ec04aaf88b9cb4882867e2b in term 1.
I20260806 02:40:44.065959 4213 leader_election.cc:302] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b19012ec04aaf88b9cb4882867e2b, 87d4bb12c08746619c0e2d8417c24794; no voters:
I20260806 02:40:44.066161 4418 raft_consensus.cc:2781] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:44.066406 4418 raft_consensus.cc:686] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 1 LEADER]: Becoming Leader. State: Replica: 213b19012ec04aaf88b9cb4882867e2b, State: Running, Role: LEADER
I20260806 02:40:44.066737 4418 consensus_queue.cc:237] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } }
I20260806 02:40:44.069754 3701 catalog_manager.cc:5495] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 213b19012ec04aaf88b9cb4882867e2b (127.28.191.3). New cstate: current_term: 1 leader_uuid: "213b19012ec04aaf88b9cb4882867e2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b19012ec04aaf88b9cb4882867e2b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:44.122514 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.121152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.28.191.3:36653)
0806 02:40:44.121276 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.121279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.121292 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:44.121392 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.121394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.121469 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.121550 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:44.122200 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:44.122203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:44.122205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:44.122216 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:44.122234 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:44.122242 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:44.122314 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:44.122372 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 02:40:44.122405 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:40:44.128324 3908 raft_consensus.cc:1261] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 1 FOLLOWER]: Refusing update from remote peer 213b19012ec04aaf88b9cb4882867e2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:44.128324 4104 raft_consensus.cc:1261] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 1 FOLLOWER]: Refusing update from remote peer 213b19012ec04aaf88b9cb4882867e2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:44.128998 4420 consensus_queue.cc:1035] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d4bb12c08746619c0e2d8417c24794" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:44.129222 4418 consensus_queue.cc:1035] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9449c423c0534feb98768ffdd9d9b6d4" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:44.136544 4425 mvcc.cc:205] Tried to move back new op lower bound from 7315390644743057408 to 7315390644499943424. Current Snapshot: MvccSnapshot[applied={T|T < 7315390644743057408}]
I20260806 02:40:44.140554 4427 mvcc.cc:205] Tried to move back new op lower bound from 7315390644743057408 to 7315390644499943424. Current Snapshot: MvccSnapshot[applied={T|T < 7315390644743057408}]
I20260806 02:40:44.145002 3698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:44.145198 3698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:44.147295 3698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0879f772940742d0aaf0aab6a6555a08: Sending DeleteTablet for 3 replicas of tablet e39f15d97e4b40aaba91245fc1cefff0
I20260806 02:40:44.147981 4069 tablet_service.cc:1514] Processing DeleteTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:44 UTC) from {username='slave'} at 127.0.0.1:55316
I20260806 02:40:44.147989 4281 tablet_service.cc:1514] Processing DeleteTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:44 UTC) from {username='slave'} at 127.0.0.1:37432
I20260806 02:40:44.148397 4437 tablet_replica.cc:330] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: stopping tablet replica
I20260806 02:40:44.148424 3880 tablet_service.cc:1514] Processing DeleteTablet for tablet e39f15d97e4b40aaba91245fc1cefff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:44 UTC) from {username='slave'} at 127.0.0.1:41674
I20260806 02:40:44.148708 4437 raft_consensus.cc:2229] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:44.148835 4438 tablet_replica.cc:330] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: stopping tablet replica
I20260806 02:40:44.148988 4437 raft_consensus.cc:2258] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:44.149112 4438 raft_consensus.cc:2229] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:44.149323 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 3786
I20260806 02:40:44.149405 4439 tablet_replica.cc:330] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4: stopping tablet replica
I20260806 02:40:44.149420 4438 raft_consensus.cc:2258] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:44.149667 4439 raft_consensus.cc:2229] T e39f15d97e4b40aaba91245fc1cefff0 P 9449c423c0534feb98768ffdd9d9b6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:44.152755 4437 ts_tablet_manager.cc:1905] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:44.155177 4438 ts_tablet_manager.cc:1905] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:44.155593 4437 ts_tablet_manager.cc:1918] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:44.155700 4437 log.cc:1196] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/e39f15d97e4b40aaba91245fc1cefff0
I20260806 02:40:44.156119 4437 ts_tablet_manager.cc:1939] T e39f15d97e4b40aaba91245fc1cefff0 P 87d4bb12c08746619c0e2d8417c24794: Deleting consensus metadata
I20260806 02:40:44.157440 3677 catalog_manager.cc:4841] TS 87d4bb12c08746619c0e2d8417c24794 (127.28.191.2:34345): tablet e39f15d97e4b40aaba91245fc1cefff0 (table test-table [id=dad3265563254416a1415cac928dca92]) successfully deleted
I20260806 02:40:44.158341 4438 ts_tablet_manager.cc:1918] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:44.158439 4438 log.cc:1196] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/e39f15d97e4b40aaba91245fc1cefff0
I20260806 02:40:44.158738 4438 ts_tablet_manager.cc:1939] T e39f15d97e4b40aaba91245fc1cefff0 P 213b19012ec04aaf88b9cb4882867e2b: Deleting consensus metadata
I20260806 02:40:44.159578 3680 catalog_manager.cc:4841] TS 213b19012ec04aaf88b9cb4882867e2b (127.28.191.3:36653): tablet e39f15d97e4b40aaba91245fc1cefff0 (table test-table [id=dad3265563254416a1415cac928dca92]) successfully deleted
W20260806 02:40:44.161019 3679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:42975 (error 108)
I20260806 02:40:44.161775 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 3978
I20260806 02:40:44.161942 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.161654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42975 (local address 127.0.0.1:41688)
0806 02:40:44.161767 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:44.161854 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:40:44.163955 3679 catalog_manager.cc:4584] TS 9449c423c0534feb98768ffdd9d9b6d4 (127.28.191.1:42975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e39f15d97e4b40aaba91245fc1cefff0: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42975: connect: Connection refused (error 111)
I20260806 02:40:44.172022 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 4186
I20260806 02:40:44.183550 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 3651
2026-08-06T02:40:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1140 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:40:44.207935 29436 mini_oidc.cc:152] Starting JWKS server
I20260806 02:40:44.209741 29436 webserver.cc:460] Webserver started at https://127.0.0.1:33265/ using document root <none> and password file <none>
I20260806 02:40:44.209888 29436 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:40:44.210110 29436 webserver.cc:460] Webserver started at http://127.0.0.1:39691/ using document root <none> and password file <none>
2026-08-06T02:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:44Z Disabled control of system clock
I20260806 02:40:44.219812 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45527
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:45527
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/testchainca.cert with env {}
W20260806 02:40:44.331636 4452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:44.331893 4452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:44.331931 4452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:44.335813 4452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:44.335867 4452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:44.335882 4452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:44.335897 4452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:44.335912 4452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:40:44.335927 4452 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:40:44.335947 4452 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33265/jwks.json
I20260806 02:40:44.341023 4452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:45527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:44.342010 4452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:44.342968 4452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:44.351859 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:44.351859 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.351966 4452 server_base.cc:1034] running on GCE node
W20260806 02:40:44.351859 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.352664 4452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:44.353835 4452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:44.355028 4452 hybrid_clock.cc:648] HybridClock initialized: now 1785984044355008 us; error 32 us; skew 500 ppm
I20260806 02:40:44.356832 4452 webserver.cc:460] Webserver started at http://127.28.191.62:38009/ using document root <none> and password file <none>
I20260806 02:40:44.357467 4452 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:44.357561 4452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:44.357795 4452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:44.359762 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "32169452bf744b18841052537461eec1"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "21b4023698f769fbaa2287003b499c91"
server_key_iv: "636282ec55cc2056ab5be796e34959e0"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.360363 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "32169452bf744b18841052537461eec1"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "21b4023698f769fbaa2287003b499c91"
server_key_iv: "636282ec55cc2056ab5be796e34959e0"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.363929 4452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:44.366326 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:44.367494 4452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:44.367631 4452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "32169452bf744b18841052537461eec1"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "21b4023698f769fbaa2287003b499c91"
server_key_iv: "636282ec55cc2056ab5be796e34959e0"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.367738 4452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:44.386914 4452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:44.387573 4452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:44.387770 4452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:44.394557 4452 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:45527
I20260806 02:40:44.394627 4543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:45527 every 8 connection(s)
I20260806 02:40:44.395509 4452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:44.395787 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 4452
I20260806 02:40:44.395924 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:44.396278 29436 external_mini_cluster.cc:1428] Setting key 0b9e281cb2dd43d18008ad2a1163b6bb
I20260806 02:40:44.398723 4544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:44.404075 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1: Bootstrap starting.
I20260806 02:40:44.405637 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.397662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.28.191.62:45527)
0806 02:40:44.398356 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.398377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.398446 (+ 69us) server_negotiation.cc:407] Connection header received
0806 02:40:44.399197 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.399215 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.399590 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.399922 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:44.400371 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.401362 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.402271 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.402733 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.402945 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.403863 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:44.403868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:44.403880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:44.404324 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:44.404379 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:44.404400 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:44.404507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:44.404680 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:40:44.404914 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260806 02:40:44.406435 4544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:44.407151 4544 log.cc:824] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:44.408422 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:40:44.409175 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1: No bootstrap required, opened a new log
I20260806 02:40:44.412699 4544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } }
I20260806 02:40:44.412943 4544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:44.413038 4544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32169452bf744b18841052537461eec1, State: Initialized, Role: FOLLOWER
I20260806 02:40:44.413451 4544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } }
I20260806 02:40:44.413601 4544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:44.413688 4544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:44.413813 4544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:44.414621 4544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } }
I20260806 02:40:44.414937 4544 leader_election.cc:302] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32169452bf744b18841052537461eec1; no voters:
I20260806 02:40:44.415253 4544 leader_election.cc:288] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:44.415374 4550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:44.415606 4550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [term 1 LEADER]: Becoming Leader. State: Replica: 32169452bf744b18841052537461eec1, State: Running, Role: LEADER
I20260806 02:40:44.415925 4550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } }
I20260806 02:40:44.416498 4544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:44.417438 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32169452bf744b18841052537461eec1. Latest consensus state: current_term: 1 leader_uuid: "32169452bf744b18841052537461eec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } } }
I20260806 02:40:44.417594 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:44.417891 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32169452bf744b18841052537461eec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32169452bf744b18841052537461eec1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45527 } } }
I20260806 02:40:44.417971 4562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:44.418004 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:44.421103 4562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:44.426352 4562 catalog_manager.cc:1347] Generated new cluster ID: 77ed01f5772e49eb997d8fac8ae581c9
I20260806 02:40:44.426419 4562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:44.435469 4562 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:44.436498 4562 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:44.444130 4562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32169452bf744b18841052537461eec1: Generated new TSK 0
I20260806 02:40:44.444790 4562 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:40:44.480846 4582 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:40:44.481010 4582 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:40:44.481647 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.457743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.28.191.62:45527)
0806 02:40:44.457861 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.457865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.458117 (+ 252us) server_negotiation.cc:407] Connection header received
0806 02:40:44.458379 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.458382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.458438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.458557 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:40:44.458817 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.459346 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.460223 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.460427 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.460556 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.460889 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:40:44.481131 (+ 20242us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:40:44.481486 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 02:40:44.481730 4548 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:40:44.483000 4581 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:44.457661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45527 (local address 127.0.0.1:55846)
0806 02:40:44.457993 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:44.458045 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:44.458215 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:44.458575 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:44.458578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:44.458586 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:40:44.458707 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:44.458713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.459465 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.459469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:44.460103 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:44.460111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.460576 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.460580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:44.460667 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.460705 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:40:44.481401 (+ 20696us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:40:44.481658 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.191.62:45527: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.191.62:45527: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:40:44.483800 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 4452
2026-08-06T02:40:44Z chronyd exiting
I20260806 02:40:44.528426 29436 test_util.cc:175] -----------------------------------------------
I20260806 02:40:44.528528 29436 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984016215184-29436-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:40:44.529795 29436 mini_oidc.cc:152] Starting JWKS server
I20260806 02:40:44.530740 29436 webserver.cc:460] Webserver started at https://127.0.0.1:35455/ using document root <none> and password file <none>
I20260806 02:40:44.530879 29436 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:40:44.531167 29436 webserver.cc:460] Webserver started at http://127.0.0.1:45381/ using document root <none> and password file <none>
2026-08-06T02:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:44Z Disabled control of system clock
I20260806 02:40:44.540946 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34337
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:34337
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35455/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/testchainca.cert with env {}
W20260806 02:40:44.651260 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:44.651623 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:44.651702 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:44.655823 4590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:44.655872 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:44.655886 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:44.655901 4590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:44.655951 4590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:40:44.655980 4590 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:40:44.656018 4590 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35455/jwks.json
I20260806 02:40:44.661278 4590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:34337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35455/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:44.662254 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:44.663240 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:44.671386 4602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:44.671379 4597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.671577 4590 server_base.cc:1034] running on GCE node
W20260806 02:40:44.671379 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.672205 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:44.673287 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:44.674469 4590 hybrid_clock.cc:648] HybridClock initialized: now 1785984044674453 us; error 29 us; skew 500 ppm
I20260806 02:40:44.680300 4590 webserver.cc:460] Webserver started at http://127.28.191.62:34191/ using document root <none> and password file <none>
I20260806 02:40:44.680912 4590 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:44.681041 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:44.681324 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:44.683351 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "b4040200fb1d4d799422078a9e95b02e"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "d7ae227190f5b1438abe5ed3548ac0c2"
server_key_iv: "f89b5c15d3b8c529e81669f0a12bb1c1"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.683957 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "b4040200fb1d4d799422078a9e95b02e"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "d7ae227190f5b1438abe5ed3548ac0c2"
server_key_iv: "f89b5c15d3b8c529e81669f0a12bb1c1"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.688571 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 02:40:44.691607 4613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:44.692734 4590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:44.692888 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "b4040200fb1d4d799422078a9e95b02e"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "d7ae227190f5b1438abe5ed3548ac0c2"
server_key_iv: "f89b5c15d3b8c529e81669f0a12bb1c1"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.693004 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:44.709585 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:44.710189 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:44.710366 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:44.716825 4590 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:34337
I20260806 02:40:44.716835 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:34337 every 8 connection(s)
I20260806 02:40:44.717720 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:44.720805 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:44.726125 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e: Bootstrap starting.
I20260806 02:40:44.726678 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 4590
I20260806 02:40:44.726815 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:44.727078 29436 external_mini_cluster.cc:1428] Setting key fd84085bbadf9b69a09474f97ea0eae8
I20260806 02:40:44.728713 4702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:44.729563 4702 log.cc:824] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:44.731825 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e: No bootstrap required, opened a new log
I20260806 02:40:44.736341 4702 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } }
I20260806 02:40:44.736608 4702 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:44.736665 4702 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4040200fb1d4d799422078a9e95b02e, State: Initialized, Role: FOLLOWER
I20260806 02:40:44.737151 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.728288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.28.191.62:34337)
0806 02:40:44.728931 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.728951 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.728993 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:40:44.729758 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.729779 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.730228 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.730572 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:44.730968 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.732424 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.733364 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.734058 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.734222 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.735000 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:44.735006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:44.735023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:44.735597 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:44.735659 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:44.735684 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:44.735777 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:44.735965 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:40:44.736214 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260806 02:40:44.737154 4702 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } }
I20260806 02:40:44.737326 4702 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:44.737404 4702 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:44.737502 4702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:44.738484 4702 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } }
I20260806 02:40:44.738855 4702 leader_election.cc:302] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4040200fb1d4d799422078a9e95b02e; no voters:
I20260806 02:40:44.739202 4702 leader_election.cc:288] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:44.739316 4707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:44.739550 4707 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [term 1 LEADER]: Becoming Leader. State: Replica: b4040200fb1d4d799422078a9e95b02e, State: Running, Role: LEADER
I20260806 02:40:44.739851 4707 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } }
I20260806 02:40:44.740028 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:40:44.740420 4702 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:44.741459 4708 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4040200fb1d4d799422078a9e95b02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } } }
I20260806 02:40:44.741578 4708 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:44.741835 4709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4040200fb1d4d799422078a9e95b02e. Latest consensus state: current_term: 1 leader_uuid: "b4040200fb1d4d799422078a9e95b02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4040200fb1d4d799422078a9e95b02e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34337 } } }
I20260806 02:40:44.741930 4709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:44.742043 4721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:44.745183 4721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:44.750497 4721 catalog_manager.cc:1347] Generated new cluster ID: caaf77580eff4925b24f135bc7a17395
I20260806 02:40:44.750569 4721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:44.756600 4721 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:44.757727 4721 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:44.769338 4721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4040200fb1d4d799422078a9e95b02e: Generated new TSK 0
I20260806 02:40:44.770051 4721 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:40:44.802501 4739 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:40:44.802605 4739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:40:44.803382 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:44.787672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.28.191.62:34337)
0806 02:40:44.787802 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:44.787809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:44.787944 (+ 135us) server_negotiation.cc:407] Connection header received
0806 02:40:44.788069 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:44.788072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:44.788126 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:44.788248 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:40:44.788440 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.789285 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.790147 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.790506 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.790601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.790900 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:40:44.802952 (+ 12052us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:40:44.803229 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:40:44.803454 4705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:40:44.803488 4738 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:44.787541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34337 (local address 127.0.0.1:51992)
0806 02:40:44.787864 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:44.787875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:44.787965 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:44.788242 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:44.788244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:44.788250 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:40:44.788344 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:44.788348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.789405 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.789408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:44.790039 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:44.790046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:44.790622 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:44.790624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:44.790690 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:44.790712 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:40:44.803171 (+ 12459us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:40:44.803318 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.191.62:34337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35455/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260806 02:40:44.804255 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 4590
2026-08-06T02:40:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:40:44.850947 29436 mini_oidc.cc:152] Starting JWKS server
I20260806 02:40:44.851811 29436 webserver.cc:460] Webserver started at https://127.0.0.1:34947/ using document root <none> and password file <none>
I20260806 02:40:44.851944 29436 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:40:44.852234 29436 webserver.cc:460] Webserver started at http://127.0.0.1:39399/ using document root <none> and password file <none>
2026-08-06T02:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:44Z Disabled control of system clock
I20260806 02:40:44.862335 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40549
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:40549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34947/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:40:44.969533 4750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:44.969785 4750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:44.969863 4750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:44.975355 4750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:44.975435 4750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:44.975462 4750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:44.975492 4750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:44.975512 4750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:40:44.975533 4750 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:40:44.975553 4750 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34947/jwks.json
I20260806 02:40:44.980355 4750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:40549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34947/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:44.981386 4750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:44.982645 4750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:44.990857 4758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:44.990816 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.990893 4750 server_base.cc:1034] running on GCE node
W20260806 02:40:44.990813 4762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:44.991694 4750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:44.992805 4750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:44.993968 4750 hybrid_clock.cc:648] HybridClock initialized: now 1785984044993932 us; error 54 us; skew 500 ppm
I20260806 02:40:44.995869 4750 webserver.cc:460] Webserver started at http://127.28.191.62:42331/ using document root <none> and password file <none>
I20260806 02:40:44.996495 4750 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:44.996557 4750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:44.996820 4750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:44.998946 4750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "0c663074e73842fdbc25972ea5024285"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "a37bceb31904eb5ce330348fec92b25c"
server_key_iv: "f2a6aea08bb9e5a810a3158cb11874f6"
server_key_version: "encryptionkey@0"
I20260806 02:40:44.999480 4750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "0c663074e73842fdbc25972ea5024285"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "a37bceb31904eb5ce330348fec92b25c"
server_key_iv: "f2a6aea08bb9e5a810a3158cb11874f6"
server_key_version: "encryptionkey@0"
I20260806 02:40:45.003070 4750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 02:40:45.005554 4770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:45.006790 4750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:45.006928 4750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "0c663074e73842fdbc25972ea5024285"
format_stamp: "Formatted at 2026-08-06 02:40:44 on dist-test-slave-tfpf"
server_key: "a37bceb31904eb5ce330348fec92b25c"
server_key_iv: "f2a6aea08bb9e5a810a3158cb11874f6"
server_key_version: "encryptionkey@0"
I20260806 02:40:45.007043 4750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:45.017527 4750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:45.018138 4750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:45.018316 4750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:45.024922 4750 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:40549
I20260806 02:40:45.024957 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:40549 every 8 connection(s)
I20260806 02:40:45.025885 4750 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:45.027447 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 4750
I20260806 02:40:45.027598 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:45.027878 29436 external_mini_cluster.cc:1428] Setting key 8951e499332ec176c91a1ea5c6b89876
I20260806 02:40:45.029381 4844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:45.034807 4844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285: Bootstrap starting.
I20260806 02:40:45.036641 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:45.029206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.28.191.62:40549)
0806 02:40:45.029824 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:45.029837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:45.029870 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:40:45.030433 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:45.030447 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:45.030808 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:45.031189 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:45.031582 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.032460 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.033465 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.033933 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.034120 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.035045 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:45.035050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:45.035061 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:45.035420 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:45.035469 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:45.035511 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:45.035618 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:45.035751 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:40:45.035985 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260806 02:40:45.037154 4844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:45.038074 4844 log.cc:824] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:45.039644 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:40:45.040180 4844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285: No bootstrap required, opened a new log
I20260806 02:40:45.043541 4844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } }
I20260806 02:40:45.043747 4844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:45.043784 4844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c663074e73842fdbc25972ea5024285, State: Initialized, Role: FOLLOWER
I20260806 02:40:45.044145 4844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } }
I20260806 02:40:45.044250 4844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:45.044293 4844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:45.044359 4844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:45.045228 4844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } }
I20260806 02:40:45.045521 4844 leader_election.cc:302] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c663074e73842fdbc25972ea5024285; no voters:
I20260806 02:40:45.045761 4844 leader_election.cc:288] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:45.045872 4854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:45.046123 4854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [term 1 LEADER]: Becoming Leader. State: Replica: 0c663074e73842fdbc25972ea5024285, State: Running, Role: LEADER
I20260806 02:40:45.046422 4854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } }
I20260806 02:40:45.047003 4844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:45.048344 4855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c663074e73842fdbc25972ea5024285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } } }
I20260806 02:40:45.048372 4857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c663074e73842fdbc25972ea5024285. Latest consensus state: current_term: 1 leader_uuid: "0c663074e73842fdbc25972ea5024285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c663074e73842fdbc25972ea5024285" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40549 } } }
I20260806 02:40:45.048544 4857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:45.048573 4855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:45.049250 4868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:45.051702 4868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:45.057358 4868 catalog_manager.cc:1347] Generated new cluster ID: a343438176a14494a585fb272cc61ec6
I20260806 02:40:45.057403 4868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:45.067137 4868 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:45.068286 4868 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:45.075554 4868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c663074e73842fdbc25972ea5024285: Generated new TSK 0
I20260806 02:40:45.076169 4868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:40:45.092228 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:45.087678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.28.191.62:40549)
0806 02:40:45.087812 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:45.087818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:45.087961 (+ 143us) server_negotiation.cc:407] Connection header received
0806 02:40:45.088118 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:45.088120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:45.088164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:45.088248 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:40:45.088519 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.088984 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.090158 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.090351 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.090427 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.090761 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:40:45.091871 (+ 1110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:40:45.092127 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 02:40:45.092307 4848 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:40:45.095075 4891 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:45.087589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:40549 (local address 127.0.0.1:40226)
0806 02:40:45.087871 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:45.087885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:45.087989 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:45.088272 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:45.088275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:45.088283 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:40:45.088404 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:45.088410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.089113 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.089117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:45.090031 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:45.090041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.090469 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.090473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:45.090565 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.090609 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:40:45.092036 (+ 1427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:40:45.094939 (+ 2903us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.191.62:40549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":2726,"thread_start_us":110,"threads_started":1}
I20260806 02:40:45.095566 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 4750
E20260806 02:40:45.113754 4892 webserver.cc:533] Webserver: sslize failed:
E20260806 02:40:45.113831 4892 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T02:40:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (298 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:45Z Disabled control of system clock
I20260806 02:40:45.159005 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43829
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:43829
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/testchainca.cert with env {}
W20260806 02:40:45.267292 4898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:45.267537 4898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:45.267575 4898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:45.271412 4898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:45.271459 4898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:45.271476 4898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:45.271492 4898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:45.271507 4898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:40:45.271521 4898 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:40:45.271534 4898 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:40:45.276779 4898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:43829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:45.277768 4898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:45.278687 4898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:45.286870 4907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:45.286887 4903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:45.286891 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:45.286986 4898 server_base.cc:1034] running on GCE node
I20260806 02:40:45.287678 4898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:45.288713 4898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:45.289896 4898 hybrid_clock.cc:648] HybridClock initialized: now 1785984045289863 us; error 47 us; skew 500 ppm
I20260806 02:40:45.291689 4898 webserver.cc:460] Webserver started at http://127.28.191.62:41499/ using document root <none> and password file <none>
I20260806 02:40:45.292265 4898 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:45.292374 4898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:45.292601 4898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:45.294710 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "cc428f988e6341a19bdc9676fe9d696d"
format_stamp: "Formatted at 2026-08-06 02:40:45 on dist-test-slave-tfpf"
server_key: "912106d2b41759950f96d5644cc07709"
server_key_iv: "ed556dea46b14e1a7245efae669cd1a8"
server_key_version: "encryptionkey@0"
I20260806 02:40:45.295295 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "cc428f988e6341a19bdc9676fe9d696d"
format_stamp: "Formatted at 2026-08-06 02:40:45 on dist-test-slave-tfpf"
server_key: "912106d2b41759950f96d5644cc07709"
server_key_iv: "ed556dea46b14e1a7245efae669cd1a8"
server_key_version: "encryptionkey@0"
I20260806 02:40:45.298991 4898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:40:45.301329 4916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:45.302459 4898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:45.302608 4898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "cc428f988e6341a19bdc9676fe9d696d"
format_stamp: "Formatted at 2026-08-06 02:40:45 on dist-test-slave-tfpf"
server_key: "912106d2b41759950f96d5644cc07709"
server_key_iv: "ed556dea46b14e1a7245efae669cd1a8"
server_key_version: "encryptionkey@0"
I20260806 02:40:45.302722 4898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:45.319550 4898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:45.320197 4898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:45.320392 4898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:45.327780 4898 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:43829
I20260806 02:40:45.328151 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:43829 every 8 connection(s)
I20260806 02:40:45.328816 4898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:45.331528 4985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:45.334427 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 4898
I20260806 02:40:45.334595 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:45.334882 29436 external_mini_cluster.cc:1428] Setting key bb0b2cf89e3d73bf25bcff4e66ea5d23
I20260806 02:40:45.337589 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d: Bootstrap starting.
I20260806 02:40:45.339680 4985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:45.340407 4985 log.cc:824] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:45.342340 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d: No bootstrap required, opened a new log
I20260806 02:40:45.344050 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:45.336260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.28.191.62:43829)
0806 02:40:45.336920 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:45.336935 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:45.336968 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:40:45.337636 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:45.337653 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:45.338059 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:45.338379 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:45.338829 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.339823 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.340780 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.341307 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.341486 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.342369 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:45.342374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:45.342386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:40:45.342801 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:45.342854 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:40:45.342875 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:45.342984 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:45.343154 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 02:40:45.343396 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260806 02:40:45.346271 4985 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } }
I20260806 02:40:45.346549 4985 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:45.346618 4985 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc428f988e6341a19bdc9676fe9d696d, State: Initialized, Role: FOLLOWER
I20260806 02:40:45.347040 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:40:45.347080 4985 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } }
I20260806 02:40:45.347256 4985 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:45.347329 4985 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:45.347445 4985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:45.348402 4985 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } }
I20260806 02:40:45.348752 4985 leader_election.cc:302] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc428f988e6341a19bdc9676fe9d696d; no voters:
I20260806 02:40:45.349136 4985 leader_election.cc:288] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:45.349246 4991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:45.349509 4991 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [term 1 LEADER]: Becoming Leader. State: Replica: cc428f988e6341a19bdc9676fe9d696d, State: Running, Role: LEADER
I20260806 02:40:45.349818 4991 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } }
I20260806 02:40:45.350351 4985 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:45.351501 4993 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc428f988e6341a19bdc9676fe9d696d. Latest consensus state: current_term: 1 leader_uuid: "cc428f988e6341a19bdc9676fe9d696d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } } }
I20260806 02:40:45.351501 4992 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc428f988e6341a19bdc9676fe9d696d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc428f988e6341a19bdc9676fe9d696d" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43829 } } }
I20260806 02:40:45.351658 4992 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:45.351658 4993 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:45.352101 5004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:45.355002 5004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:45.360558 5004 catalog_manager.cc:1347] Generated new cluster ID: 7d007eb4172148c4a559b57c2e79cb6f
I20260806 02:40:45.360637 5004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:45.370497 5004 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:45.371312 5004 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:45.377653 5004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc428f988e6341a19bdc9676fe9d696d: Generated new TSK 0
I20260806 02:40:45.378285 5004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:40:45.868695 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:45.390047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.28.191.62:43829)
0806 02:40:45.390192 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:45.390197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:45.390417 (+ 220us) server_negotiation.cc:407] Connection header received
0806 02:40:45.390584 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:45.390587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:45.390634 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:45.390721 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:40:45.390985 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.391499 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.392719 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.392935 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.393011 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.393380 (+ 369us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:40:45.868283 (+474903us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:40:45.868570 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60338: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:40:45.868774 4988 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60338: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:40:45.868973 5023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:45.389995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43829 (local address 127.0.0.1:60338)
0806 02:40:45.390320 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:45.390334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:45.390452 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:45.390744 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:45.390747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:45.390758 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:40:45.390876 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:45.390882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.391603 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.391607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:45.392596 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:45.392606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:45.393091 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:45.393096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:45.393193 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:45.393222 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:40:45.868527 (+475305us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:40:45.868720 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.191.62:43829: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":136,"threads_started":1}
I20260806 02:40:45.869678 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 4898
2026-08-06T02:40:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (744 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:46 dist-test-slave-tfpf krb5kdc[5041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:46 dist-test-slave-tfpf krb5kdc[5041](info): set up 2 sockets
Aug 06 02:40:46 dist-test-slave-tfpf krb5kdc[5041](info): commencing operation
krb5kdc: starting...
W20260806 02:40:47.307595 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:47 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:40:47.430189 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36041
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:47.538749 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:47.539026 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:47.539126 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:47.543004 5066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:47.543099 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:47.543136 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:47.543212 5066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:47.543272 5066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:47.548182 5066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:47.549255 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:47.550194 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:47.558532 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:47.558526 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:47.558522 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:47.558560 5066 server_base.cc:1034] running on GCE node
I20260806 02:40:47.559345 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:47.560375 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:47.561551 5066 hybrid_clock.cc:648] HybridClock initialized: now 1785984047561526 us; error 39 us; skew 500 ppm
Aug 06 02:40:47 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:47.564370 5066 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:40:47.565282 5066 webserver.cc:460] Webserver started at http://127.28.191.62:37681/ using document root <none> and password file <none>
I20260806 02:40:47.565793 5066 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:47.565877 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:47.566097 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:47.568011 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "fb87f01b24584bfc83b610d7bf734fa8"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "80a8f87a3dd0f34b108e6cadba14fa40"
server_key_iv: "5f37bc1a4522cd7f817c2ef2e2a8a588"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.568574 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "fb87f01b24584bfc83b610d7bf734fa8"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "80a8f87a3dd0f34b108e6cadba14fa40"
server_key_iv: "5f37bc1a4522cd7f817c2ef2e2a8a588"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.572127 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:40:47.574496 5086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:47.575582 5066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:47.575706 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "fb87f01b24584bfc83b610d7bf734fa8"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "80a8f87a3dd0f34b108e6cadba14fa40"
server_key_iv: "5f37bc1a4522cd7f817c2ef2e2a8a588"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.575802 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:47.591274 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:47.593966 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:47.594161 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:47.600787 5066 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36041
I20260806 02:40:47.600838 5158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36041 every 8 connection(s)
I20260806 02:40:47.601707 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:47.604328 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:47.605903 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5066
I20260806 02:40:47.606060 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:47.606313 29436 external_mini_cluster.cc:1428] Setting key aa82d25017fad9613aa44687903ed06a
I20260806 02:40:47.610844 5159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8: Bootstrap starting.
I20260806 02:40:47.613433 5159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:47.614320 5159 log.cc:824] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:40:47 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:47.616447 5159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8: No bootstrap required, opened a new log
I20260806 02:40:47.621167 5159 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } }
I20260806 02:40:47.621379 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:47.607634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.28.191.62:36041)
0806 02:40:47.608034 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:47.608052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:47.608094 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:40:47.608999 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:47.609022 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:47.609433 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:47.609793 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:47.610448 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.611866 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.612709 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.613415 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.613595 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:47.616342 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:47.616348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:47.616362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:47.616390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:47.618324 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:47.618733 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:47.618740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:47.618748 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:47.618908 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:47.619186 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:47.619189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:47.619191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:47.619712 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:47.619905 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:47.620147 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:40:47.620442 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260806 02:40:47.621423 5159 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:47.621469 5159 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb87f01b24584bfc83b610d7bf734fa8, State: Initialized, Role: FOLLOWER
I20260806 02:40:47.621907 5159 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } }
I20260806 02:40:47.622071 5159 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:47.622138 5159 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:47.622236 5159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:47.623090 5159 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } }
I20260806 02:40:47.623421 5159 leader_election.cc:302] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb87f01b24584bfc83b610d7bf734fa8; no voters:
I20260806 02:40:47.623739 5159 leader_election.cc:288] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:47.623847 5165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:47.624092 5165 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [term 1 LEADER]: Becoming Leader. State: Replica: fb87f01b24584bfc83b610d7bf734fa8, State: Running, Role: LEADER
I20260806 02:40:47.624400 5165 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } }
I20260806 02:40:47.624920 5159 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:47.626114 5166 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb87f01b24584bfc83b610d7bf734fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } } }
I20260806 02:40:47.626114 5167 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb87f01b24584bfc83b610d7bf734fa8. Latest consensus state: current_term: 1 leader_uuid: "fb87f01b24584bfc83b610d7bf734fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb87f01b24584bfc83b610d7bf734fa8" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36041 } } }
I20260806 02:40:47.626276 5166 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:47.626276 5167 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:47.626664 5177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:47.629418 5177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:47.634362 5177 catalog_manager.cc:1347] Generated new cluster ID: 88dd508654784c07878116bccbc7035a
I20260806 02:40:47.634421 5177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:47.659565 5177 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:47.660830 5177 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:47.668977 5177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb87f01b24584bfc83b610d7bf734fa8: Generated new TSK 0
I20260806 02:40:47.669780 5177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:40:47.704612 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:47.815361 5192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:47.815665 5192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:47.815747 5192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:47.819841 5192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:47.819933 5192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:47.820032 5192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:47.825150 5192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:47.826283 5192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:47.827368 5192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:47.836858 5202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:47.836957 5198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:47.836853 5199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:47.837160 5192 server_base.cc:1034] running on GCE node
I20260806 02:40:47.837704 5192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:47.838356 5192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:47.839541 5192 hybrid_clock.cc:648] HybridClock initialized: now 1785984047839533 us; error 31 us; skew 500 ppm
Aug 06 02:40:47 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:47.842332 5192 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:40:47.843142 5192 webserver.cc:460] Webserver started at http://127.28.191.1:37563/ using document root <none> and password file <none>
I20260806 02:40:47.843667 5192 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:47.843724 5192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:47.843886 5192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:47.846184 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "7667c040346548bf817cfbbf33723c36"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "6c836f408c18691f88ead8cd3a57f478"
server_key_iv: "33aeba7b3a1b4c8908a96b2fc6a472f3"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.846705 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "7667c040346548bf817cfbbf33723c36"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "6c836f408c18691f88ead8cd3a57f478"
server_key_iv: "33aeba7b3a1b4c8908a96b2fc6a472f3"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.850136 5192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:47.852304 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:47.853493 5192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:47.853600 5192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7667c040346548bf817cfbbf33723c36"
format_stamp: "Formatted at 2026-08-06 02:40:47 on dist-test-slave-tfpf"
server_key: "6c836f408c18691f88ead8cd3a57f478"
server_key_iv: "33aeba7b3a1b4c8908a96b2fc6a472f3"
server_key_version: "encryptionkey@0"
I20260806 02:40:47.853681 5192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:47.864006 5192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:47.866660 5192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:47.866803 5192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:47.867282 5192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:47.868002 5192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:47.868055 5192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:47.868094 5192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:47.868124 5192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:47.876736 5192 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:34479
I20260806 02:40:47.876753 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:34479 every 8 connection(s)
I20260806 02:40:47.877663 5192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:47.881172 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5192
I20260806 02:40:47.881316 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:47.881670 29436 external_mini_cluster.cc:1428] Setting key 46a9456aa6324335a2c0f2e7107dde52
Aug 06 02:40:47 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:47.892606 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:47.879501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:54677 (local address 127.28.191.62:36041)
0806 02:40:47.879665 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:47.879669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:47.880478 (+ 809us) server_negotiation.cc:407] Connection header received
0806 02:40:47.882263 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:47.882267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:47.882323 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:47.882391 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:47.883844 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.884359 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.885851 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.886250 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.886369 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:47.888984 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:47.888987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:47.888990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:47.889018 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:47.890563 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:47.891079 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:47.891083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:47.891085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:47.891180 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:47.891511 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:47.891516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:47.891518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:47.891696 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:47.891801 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:47.892315 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 02:40:47.892438 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:40:47.893496 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:47.879913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36041 (local address 127.28.191.1:54677)
0806 02:40:47.880334 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:47.880374 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:47.882016 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:47.882583 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:47.882590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:47.883001 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:47.883687 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:47.883711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.884495 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.884501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:47.885730 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:47.885740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:47.886394 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:47.886397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:47.886526 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:47.887110 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:47.887135 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:47.888826 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:47.890702 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:47.890708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:47.890721 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:47.890976 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:47.891274 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:47.891277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:47.891279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:47.891416 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:47.891801 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:47.891808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:47.892042 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:40:47.892275 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 02:40:47.892575 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:40:47.894785 5382 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36041
I20260806 02:40:47.895035 5382 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:47.895622 5382 heartbeater.cc:511] Master 127.28.191.62:36041 requested a full tablet report, sending...
I20260806 02:40:47.897267 5108 ts_manager.cc:194] Registered new tserver with Master: 7667c040346548bf817cfbbf33723c36 (127.28.191.1:34479)
I20260806 02:40:47.898391 5108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:54677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:40:47.909768 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:48.019408 5390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:48.019699 5390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:48.019779 5390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:48.023720 5390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:48.023802 5390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:48.023886 5390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:48.029033 5390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:48.030011 5390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:48.030957 5390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:48.040830 5398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:48.040835 5400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:48.041082 5390 server_base.cc:1034] running on GCE node
W20260806 02:40:48.040858 5397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:48.041617 5390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:48.042218 5390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:48.043385 5390 hybrid_clock.cc:648] HybridClock initialized: now 1785984048043346 us; error 53 us; skew 500 ppm
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.046288 5390 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:40:48.047137 5390 webserver.cc:460] Webserver started at http://127.28.191.2:38025/ using document root <none> and password file <none>
I20260806 02:40:48.047691 5390 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:48.047752 5390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:48.047997 5390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:48.050071 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "c503056fb62941cfaa7619a4a460dfa1"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "1a418ac5f13663654605daed9477c98c"
server_key_iv: "918b888009392c24aeddd3b39883e688"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.050630 5390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "c503056fb62941cfaa7619a4a460dfa1"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "1a418ac5f13663654605daed9477c98c"
server_key_iv: "918b888009392c24aeddd3b39883e688"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.054203 5390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:48.056617 5409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.057910 5390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:48.058048 5390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "c503056fb62941cfaa7619a4a460dfa1"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "1a418ac5f13663654605daed9477c98c"
server_key_iv: "918b888009392c24aeddd3b39883e688"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.058161 5390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:48.081243 5390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:48.084581 5390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:48.084787 5390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:48.085455 5390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:48.086292 5390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:48.086362 5390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.086431 5390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:48.086483 5390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.095114 5390 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:33009
I20260806 02:40:48.095222 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:33009 every 8 connection(s)
I20260806 02:40:48.095999 5390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:48.106226 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5390
I20260806 02:40:48.106371 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:48.106673 29436 external_mini_cluster.cc:1428] Setting key 306ba0efdb1c494f6c2ff0c7be5de3a6
I20260806 02:40:48.110127 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.098036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:42501 (local address 127.28.191.62:36041)
0806 02:40:48.098167 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.098171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.098950 (+ 779us) server_negotiation.cc:407] Connection header received
0806 02:40:48.100388 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.100392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.100451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.100583 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.101779 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.102306 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.103567 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.103787 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.103897 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.106371 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:48.106374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:48.106376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:48.106398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:48.107948 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.108500 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.108507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.108510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.108594 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.108945 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.108950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.108952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.109218 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:48.109327 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:48.109815 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 02:40:48.109947 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:40:48.110888 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.098349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36041 (local address 127.28.191.2:42501)
0806 02:40:48.098797 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:48.098849 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:48.100177 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:48.100751 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:48.100757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:48.101168 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:48.101637 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:48.101651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.102437 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.102446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:48.103422 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:48.103432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.103938 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.103942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:48.104115 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.104696 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:48.104712 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:48.106194 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:48.108079 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:48.108085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:48.108099 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:48.108393 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:48.108691 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:48.108694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:48.108696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:48.108845 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:48.109327 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:48.109334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:48.109579 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 02:40:48.109771 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 02:40:48.110079 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260806 02:40:48.112108 5572 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36041
I20260806 02:40:48.112346 5572 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:48.112809 5572 heartbeater.cc:511] Master 127.28.191.62:36041 requested a full tablet report, sending...
I20260806 02:40:48.113876 5108 ts_manager.cc:194] Registered new tserver with Master: c503056fb62941cfaa7619a4a460dfa1 (127.28.191.2:33009)
I20260806 02:40:48.114465 5108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:42501
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:40:48.134289 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:48.243167 5580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:48.243404 5580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:48.243444 5580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:48.247427 5580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:48.247476 5580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:48.247576 5580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:48.252519 5580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:48.253490 5580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:48.254429 5580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:48.263906 5587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:48.263916 5589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:48.263923 5586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:48.263960 5580 server_base.cc:1034] running on GCE node
I20260806 02:40:48.264519 5580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:48.265210 5580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:48.266368 5580 hybrid_clock.cc:648] HybridClock initialized: now 1785984048266352 us; error 29 us; skew 500 ppm
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.269225 5580 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:40:48.270066 5580 webserver.cc:460] Webserver started at http://127.28.191.3:33017/ using document root <none> and password file <none>
I20260806 02:40:48.270620 5580 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:48.270710 5580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:48.270923 5580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:48.272888 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "a336da21a3f448368140a1ba3e2efb66"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "7a4c77f7ed75c54c26fe325fda043a1b"
server_key_iv: "780b1d1b9e167e4f0c7916fe343ab02b"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.273517 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "a336da21a3f448368140a1ba3e2efb66"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "7a4c77f7ed75c54c26fe325fda043a1b"
server_key_iv: "780b1d1b9e167e4f0c7916fe343ab02b"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.277179 5580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:40:48.279547 5598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.280771 5580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:48.280917 5580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "a336da21a3f448368140a1ba3e2efb66"
format_stamp: "Formatted at 2026-08-06 02:40:48 on dist-test-slave-tfpf"
server_key: "7a4c77f7ed75c54c26fe325fda043a1b"
server_key_iv: "780b1d1b9e167e4f0c7916fe343ab02b"
server_key_version: "encryptionkey@0"
I20260806 02:40:48.281049 5580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:48.290612 5580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:48.293802 5580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:48.294015 5580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:48.294580 5580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:48.295358 5580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:48.295429 5580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.295496 5580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:48.295544 5580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:48.304304 5580 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:41701
I20260806 02:40:48.304344 5762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:41701 every 8 connection(s)
I20260806 02:40:48.305156 5580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:48.310154 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5580
I20260806 02:40:48.310307 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:48.310642 29436 external_mini_cluster.cc:1428] Setting key 50665dddc75fef660cd41875f02e1031
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:40:48.318614 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.306842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42905 (local address 127.28.191.62:36041)
0806 02:40:48.306966 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.306969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.307711 (+ 742us) server_negotiation.cc:407] Connection header received
0806 02:40:48.309129 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.309133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.309194 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.309274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.310437 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.311108 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.312602 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.312896 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.312979 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.315277 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:48.315280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:48.315282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:48.315304 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:48.316774 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.317326 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.317329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.317330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.317403 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.317704 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.317706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.317707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.317863 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:48.317938 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:48.318411 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 02:40:48.318503 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:40:48.319365 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.307121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36041 (local address 127.28.191.3:42905)
0806 02:40:48.307555 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:48.307595 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:48.308845 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:48.309432 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:48.309438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:48.309762 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:48.310209 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:48.310221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.311240 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.311244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:48.312451 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:48.312462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.313033 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.313037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:48.313245 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.313735 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:48.313750 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:48.315132 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:48.316882 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:48.316887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:48.316897 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:48.317238 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:48.317495 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:48.317498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:48.317499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:48.317625 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:48.317951 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:48.317967 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:48.318199 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 02:40:48.318344 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:40:48.318602 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260806 02:40:48.320501 5763 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36041
I20260806 02:40:48.320696 5763 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:48.321204 5763 heartbeater.cc:511] Master 127.28.191.62:36041 requested a full tablet report, sending...
I20260806 02:40:48.322209 5108 ts_manager.cc:194] Registered new tserver with Master: a336da21a3f448368140a1ba3e2efb66 (127.28.191.3:41701)
I20260806 02:40:48.322717 5108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:42905
I20260806 02:40:48.324676 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/test-admin.keytab.
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.341009 29436 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:40:48.344182 5770 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:48.342511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.28.191.1:34479)
0806 02:40:48.342922 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.342933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.342954 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:40:48.343005 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.343011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.343196 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.343348 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.343977 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58558: BlockingRecv error: recv got EOF from 127.0.0.1:58558 (error 108)
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":146,"threads_started":1}
W20260806 02:40:48.344260 29436 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.345530 29436 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.346601 29436 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:48.356479 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.347506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.28.191.1:34479)
0806 02:40:48.347644 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.347647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.347675 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:40:48.347791 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.347793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.347837 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.347905 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.348404 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.349277 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.350156 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.350503 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.350590 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.352941 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:48.352950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:48.352962 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:48.352985 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:48.354746 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.355175 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.355181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.355182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.355258 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.355500 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.355501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.355503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.355797 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:48.356005 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:48.356143 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:40:48.356323 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 02:40:48.360096 5770 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:48.359241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.28.191.1:34479)
0806 02:40:48.359352 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.359356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.359471 (+ 115us) server_negotiation.cc:407] Connection header received
0806 02:40:48.359566 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.359568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.359600 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.359664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.359977 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58576: BlockingRecv error: recv got EOF from 127.0.0.1:58576 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 02:40:48.360526 29436 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc)
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.361685 29436 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.362600 29436 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:48.370797 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.363445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.28.191.1:34479)
0806 02:40:48.363562 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.363566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.363680 (+ 114us) server_negotiation.cc:407] Connection header received
0806 02:40:48.363794 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.363797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.363838 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.363919 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.364422 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.364904 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.365700 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.365914 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.366045 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.368062 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:48.368065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:48.368067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:48.368088 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:48.369508 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.369881 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.369884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.369885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.369953 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.370171 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.370173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.370174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.370327 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:48.370417 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:48.370612 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 02:40:48.370700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:40:48.373672 5770 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:48.372377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.28.191.1:34479)
0806 02:40:48.372563 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.372569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.372705 (+ 136us) server_negotiation.cc:407] Connection header received
0806 02:40:48.372901 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.372905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.372953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.373069 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.373560 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58592: BlockingRecv error: recv got EOF from 127.0.0.1:58592 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260806 02:40:48.373800 29436 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984016215184-29436-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.374725 29436 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:48.375583 29436 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:40:48.384244 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:48.376295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.28.191.1:34479)
0806 02:40:48.376433 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:48.376437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:48.376549 (+ 112us) server_negotiation.cc:407] Connection header received
0806 02:40:48.376686 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:48.376688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:48.376740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:48.376823 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:48.377393 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.377863 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.378636 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:48.378912 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:48.379019 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:48.381245 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:48.381249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:48.381251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:48.381281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:48.382669 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.383089 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.383094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.383096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.383177 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:48.383461 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:48.383465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:48.383466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:48.383625 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:48.383769 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:48.384005 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:40:48.384109 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:40:48.385494 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5192
I20260806 02:40:48.398620 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5390
I20260806 02:40:48.408073 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5580
I20260806 02:40:48.414510 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5066
2026-08-06T02:40:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2300 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5791](info): set up 2 sockets
Aug 06 02:40:48 dist-test-slave-tfpf krb5kdc[5791](info): commencing operation
krb5kdc: starting...
W20260806 02:40:49.617040 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:49 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab.
Entry for principal oryx/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab.
I20260806 02:40:49.741851 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35097
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab
--principal=oryx/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:49.850697 5808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:49.850955 5808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:49.851035 5808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:49.854936 5808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:49.855007 5808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:49.855046 5808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:49.855121 5808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:49.855173 5808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:49.860178 5808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab
--principal=oryx/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:49.861194 5808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:49.862138 5808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:49.870260 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:49.870287 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:49.870271 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:49.870493 5808 server_base.cc:1034] running on GCE node
I20260806 02:40:49.871165 5808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:49.872202 5808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:49.873361 5808 hybrid_clock.cc:648] HybridClock initialized: now 1785984049873335 us; error 39 us; skew 500 ppm
Aug 06 02:40:49 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:49.881349 5808 init.cc:377] Logged in from keytab as oryx/127.28.191.62@KRBTEST.COM (short username oryx)
I20260806 02:40:49.882313 5808 webserver.cc:460] Webserver started at http://127.28.191.62:40569/ using document root <none> and password file <none>
I20260806 02:40:49.882872 5808 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:49.882934 5808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:49.883193 5808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:49.885253 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "b10b88d1229a41088fd2232e3a46a931"
format_stamp: "Formatted at 2026-08-06 02:40:49 on dist-test-slave-tfpf"
server_key: "e16b8ef5b55f0d9fcbc11f9d1170930b"
server_key_iv: "ba25abe804f2186de396df70d812c7a7"
server_key_version: "encryptionkey@0"
I20260806 02:40:49.885841 5808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "b10b88d1229a41088fd2232e3a46a931"
format_stamp: "Formatted at 2026-08-06 02:40:49 on dist-test-slave-tfpf"
server_key: "e16b8ef5b55f0d9fcbc11f9d1170930b"
server_key_iv: "ba25abe804f2186de396df70d812c7a7"
server_key_version: "encryptionkey@0"
I20260806 02:40:49.889400 5808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:49.891793 5831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:49.892903 5808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:49.893049 5808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "b10b88d1229a41088fd2232e3a46a931"
format_stamp: "Formatted at 2026-08-06 02:40:49 on dist-test-slave-tfpf"
server_key: "e16b8ef5b55f0d9fcbc11f9d1170930b"
server_key_iv: "ba25abe804f2186de396df70d812c7a7"
server_key_version: "encryptionkey@0"
I20260806 02:40:49.893193 5808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:49.913841 5808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:49.916782 5808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:49.917016 5808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:49.924481 5808 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35097
I20260806 02:40:49.924763 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35097 every 8 connection(s)
I20260806 02:40:49.925654 5808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:49.928448 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5808
I20260806 02:40:49.928591 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:49.928627 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:49.928923 29436 external_mini_cluster.cc:1428] Setting key cb41a4df9f7527b5e1eb35b73b5ab921
I20260806 02:40:49.935487 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Bootstrap starting.
I20260806 02:40:49.938072 5898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Neither blocks nor log segments found. Creating new log.
Aug 06 02:40:49 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:49.938854 5898 log.cc:824] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:49.940786 5898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: No bootstrap required, opened a new log
I20260806 02:40:49.943894 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:49.930354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.28.191.62:35097)
0806 02:40:49.930806 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:49.930835 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:49.930872 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:40:49.931837 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:49.931858 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:49.932187 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:49.932503 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:49.933449 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:49.934430 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:49.935672 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:49.936334 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:49.936498 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:49.939188 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:49.939193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:49.939206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:49.939232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:49.941005 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:49.941564 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:49.941571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:49.941581 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:49.941742 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:49.942080 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:49.942082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:49.942084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:49.942445 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:49.942593 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:49.942884 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 02:40:49.943176 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":156,"threads_started":1}
I20260806 02:40:49.944943 5898 raft_consensus.cc:348] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } }
I20260806 02:40:49.945238 5898 raft_consensus.cc:374] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:49.945298 5898 raft_consensus.cc:729] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10b88d1229a41088fd2232e3a46a931, State: Initialized, Role: FOLLOWER
I20260806 02:40:49.945730 5898 consensus_queue.cc:260] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } }
I20260806 02:40:49.945941 5898 raft_consensus.cc:388] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:49.946074 5898 raft_consensus.cc:482] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:49.946199 5898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:49.947075 5898 raft_consensus.cc:504] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } }
I20260806 02:40:49.947400 5898 leader_election.cc:302] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b10b88d1229a41088fd2232e3a46a931; no voters:
I20260806 02:40:49.947705 5898 leader_election.cc:288] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:49.947877 5903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:49.948113 5903 raft_consensus.cc:686] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 LEADER]: Becoming Leader. State: Replica: b10b88d1229a41088fd2232e3a46a931, State: Running, Role: LEADER
I20260806 02:40:49.948421 5903 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } }
I20260806 02:40:49.948876 5898 sys_catalog.cc:564] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:49.949892 5906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b10b88d1229a41088fd2232e3a46a931. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } }
I20260806 02:40:49.950018 5906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:49.950389 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } }
I20260806 02:40:49.950497 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:49.951568 5915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:49.954149 5915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:49.959107 5915 catalog_manager.cc:1347] Generated new cluster ID: 1d2d2c184bb645ec80bf9ebfef440b71
I20260806 02:40:49.959224 5915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:49.981379 5915 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:49.982590 5915 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:49.988039 5915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Generated new TSK 0
I20260806 02:40:49.988817 5915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
I20260806 02:40:50.021361 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:50.132025 5930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:50.132265 5930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:50.132306 5930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:50.136291 5930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:50.136340 5930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:50.136425 5930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:50.141511 5930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:50.142516 5930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:50.143486 5930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:50.152805 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.152812 5947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.152879 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:50.153026 5930 server_base.cc:1034] running on GCE node
I20260806 02:40:50.153458 5930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:50.154039 5930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:50.155226 5930 hybrid_clock.cc:648] HybridClock initialized: now 1785984050155214 us; error 27 us; skew 500 ppm
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:50.157999 5930 init.cc:377] Logged in from keytab as oryx/127.28.191.1@KRBTEST.COM (short username oryx)
I20260806 02:40:50.158905 5930 webserver.cc:460] Webserver started at http://127.28.191.1:32889/ using document root <none> and password file <none>
I20260806 02:40:50.159441 5930 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:50.159529 5930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:50.159758 5930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:50.161720 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "7975b79b3b5d4bf7b27b0c11fedef25e"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c5def424cd6a5113c67901c983b28e35"
server_key_iv: "0ed2995a1767cd355dc5092c5e9dd0b1"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.162258 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "7975b79b3b5d4bf7b27b0c11fedef25e"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c5def424cd6a5113c67901c983b28e35"
server_key_iv: "0ed2995a1767cd355dc5092c5e9dd0b1"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.165725 5930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:50.167919 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.169246 5930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:50.169394 5930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7975b79b3b5d4bf7b27b0c11fedef25e"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c5def424cd6a5113c67901c983b28e35"
server_key_iv: "0ed2995a1767cd355dc5092c5e9dd0b1"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.169509 5930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:50.197754 5930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:50.201015 5930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:50.201252 5930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:50.201874 5930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:50.202668 5930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:50.202737 5930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.202807 5930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:50.202848 5930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.211894 5930 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42503
I20260806 02:40:50.211938 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42503 every 8 connection(s)
I20260806 02:40:50.212726 5930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:50.218083 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 5930
I20260806 02:40:50.218199 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:50.218484 29436 external_mini_cluster.cc:1428] Setting key eff4de0ee7407b39ec532be3a998a41f
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:50.226531 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.214422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:55531 (local address 127.28.191.62:35097)
0806 02:40:50.214549 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.214554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.215334 (+ 780us) server_negotiation.cc:407] Connection header received
0806 02:40:50.216664 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.216667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.216739 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.216854 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.218024 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.218658 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.219708 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.219949 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.220058 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.222433 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.222438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.222441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.222472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.224087 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.224756 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.224759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.224760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.224846 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.225266 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.225268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.225269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.225428 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.225512 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.226156 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.226301 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:40:50.227039 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.214761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:55531)
0806 02:40:50.215195 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:50.215235 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:50.216456 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:50.217030 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:50.217037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:50.217438 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:50.217868 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.217880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.218805 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.218809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.219587 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.219596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.220099 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.220103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.220263 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.220813 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:50.220827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:50.222283 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:50.224208 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.224217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.224233 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.224649 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.224935 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.224941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.224945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.225152 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.225552 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:50.225560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:50.225898 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 02:40:50.226048 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:40:50.226264 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260806 02:40:50.228260 6104 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:50.228494 6104 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:50.228935 6104 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:50.230419 5851 ts_manager.cc:194] Registered new tserver with Master: 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503)
I20260806 02:40:50.231535 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:55531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
I20260806 02:40:50.247594 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:50.359494 6111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:50.359737 6111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:50.359781 6111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:50.363759 6111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:50.363809 6111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:50.363865 6111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:50.368958 6111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:50.369926 6111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:50.370893 6111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:50.380337 6117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.380352 6120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.380352 6118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:50.380554 6111 server_base.cc:1034] running on GCE node
I20260806 02:40:50.381124 6111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:50.381767 6111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:50.382972 6111 hybrid_clock.cc:648] HybridClock initialized: now 1785984050382948 us; error 35 us; skew 500 ppm
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:50.385658 6111 init.cc:377] Logged in from keytab as oryx/127.28.191.2@KRBTEST.COM (short username oryx)
I20260806 02:40:50.386534 6111 webserver.cc:460] Webserver started at http://127.28.191.2:41481/ using document root <none> and password file <none>
I20260806 02:40:50.387104 6111 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:50.387197 6111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:50.387431 6111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:50.389797 6111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "bc2a8490f83a4a3889ae7e5193e23922"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c65aac5343631d37faf9346e1444930f"
server_key_iv: "78619823967dd166352151317054b84a"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.390367 6111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "bc2a8490f83a4a3889ae7e5193e23922"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c65aac5343631d37faf9346e1444930f"
server_key_iv: "78619823967dd166352151317054b84a"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.394106 6111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:40:50.396410 6130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.397814 6111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:50.397938 6111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "bc2a8490f83a4a3889ae7e5193e23922"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c65aac5343631d37faf9346e1444930f"
server_key_iv: "78619823967dd166352151317054b84a"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.398046 6111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:50.409386 6111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:50.412186 6111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:50.412392 6111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:50.412954 6111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:50.413851 6111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:50.413921 6111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.413986 6111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:50.414021 6111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.423269 6111 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:42979
I20260806 02:40:50.423287 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:42979 every 8 connection(s)
I20260806 02:40:50.424198 6111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:50.434379 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 6111
I20260806 02:40:50.434537 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:50.434856 29436 external_mini_cluster.cc:1428] Setting key ec7086796949371dd0d31e443e6eb925
I20260806 02:40:50.438445 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.426033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:42511 (local address 127.28.191.62:35097)
0806 02:40:50.426208 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.426213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.426995 (+ 782us) server_negotiation.cc:407] Connection header received
0806 02:40:50.428351 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.428354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.428401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.428473 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.429689 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.430203 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.431323 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.431533 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.431608 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.434184 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.434188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.434192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.434221 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.435948 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.436604 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.436607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.436609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.436705 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.437185 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.437191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.437195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.437459 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.437573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.438055 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.438200 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:40:50.439272 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.426394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:42511)
0806 02:40:50.426832 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:50.426874 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:50.428140 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:50.428645 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:50.428651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:50.428997 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:50.429553 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.429567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.430313 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.430315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.431210 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.431220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.431663 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.431666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.431809 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.432440 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:50.432457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:50.433977 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:50.436074 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.436080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.436095 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.436495 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.436845 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.436847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.436848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.437014 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.437575 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:50.437580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:50.437801 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 02:40:50.437957 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 02:40:50.438229 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260806 02:40:50.440497 6282 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:50.440733 6282 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:50.441267 6282 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
I20260806 02:40:50.442363 5851 ts_manager.cc:194] Registered new tserver with Master: bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:50.442961 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:42511
WARNING: no policy specified for oryx/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
I20260806 02:40:50.464284 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:50.576854 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:50.577149 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:50.577236 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:50.581041 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:50.581142 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:50.581243 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:50.586254 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:50.587289 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:50.588248 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:50.597643 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.597643 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.597702 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:50.597707 6290 server_base.cc:1034] running on GCE node
I20260806 02:40:50.598321 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:50.598963 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:50.600148 6290 hybrid_clock.cc:648] HybridClock initialized: now 1785984050600121 us; error 48 us; skew 500 ppm
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:50.603053 6290 init.cc:377] Logged in from keytab as oryx/127.28.191.3@KRBTEST.COM (short username oryx)
I20260806 02:40:50.603838 6290 webserver.cc:460] Webserver started at http://127.28.191.3:33897/ using document root <none> and password file <none>
I20260806 02:40:50.604341 6290 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:50.604398 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:50.604568 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:50.606526 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "25afa3a9cdf343b3977affe8f30b3ee8"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "35ab5e8e3c9c0c76482ccbf801027f74"
server_key_iv: "0ab458e234913a4fc61195c74e3b8b50"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.607046 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "25afa3a9cdf343b3977affe8f30b3ee8"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "35ab5e8e3c9c0c76482ccbf801027f74"
server_key_iv: "0ab458e234913a4fc61195c74e3b8b50"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.611483 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:40:50.615105 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.617038 6290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:50.617274 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "25afa3a9cdf343b3977affe8f30b3ee8"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "35ab5e8e3c9c0c76482ccbf801027f74"
server_key_iv: "0ab458e234913a4fc61195c74e3b8b50"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.617425 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:50.645876 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:50.649133 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:50.649331 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:50.650033 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:50.650899 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:50.650967 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.651039 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:50.651088 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.659600 6290 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:38491
I20260806 02:40:50.659626 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:38491 every 8 connection(s)
I20260806 02:40:50.660455 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:50.661468 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 6290
I20260806 02:40:50.661592 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:50.661916 29436 external_mini_cluster.cc:1428] Setting key 1f8174a416b6265c6206e1d22b28555e
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:50.674758 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.662512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:41909 (local address 127.28.191.62:35097)
0806 02:40:50.662680 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.662684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.663620 (+ 936us) server_negotiation.cc:407] Connection header received
0806 02:40:50.665018 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.665021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.665130 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.665212 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.666359 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.666885 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.668104 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.668314 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.668394 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.670946 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.670949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.670952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.670975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.672581 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.673131 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.673134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.673136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.673212 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.673590 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.673595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.673596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.673812 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.673925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.674457 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.674587 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:40:50.675797 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.662989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:41909)
0806 02:40:50.663485 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:50.663539 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:50.664836 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:50.665381 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:50.665388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:50.665739 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:50.666220 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.666233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.666993 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.666995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.667998 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.668008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.668439 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.668441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.668581 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.669250 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:50.669277 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:50.670787 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:50.672720 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.672728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.672745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.672999 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.673323 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.673325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.673327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.673478 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.673940 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:50.673949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:50.674198 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 02:40:50.674396 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 02:40:50.674725 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260806 02:40:50.676966 6472 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:50.677240 6472 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:50.677721 6472 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
I20260806 02:40:50.678725 5851 ts_manager.cc:194] Registered new tserver with Master: 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:50.679302 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:41909
I20260806 02:40:50.688009 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.191.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.61.keytab.
Entry for principal oryx/127.28.191.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.61.keytab.
I20260806 02:40:50.718187 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:44167
--webserver_interface=127.28.191.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.61.keytab
--principal=oryx/127.28.191.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35097,127.28.191.61:44167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:50.831070 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:50.831327 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:50.831367 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:50.835253 6477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:50.835302 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:50.835320 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:50.835336 6477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:50.835351 6477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:50.840487 6477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35097,127.28.191.61:44167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.61.keytab
--principal=oryx/127.28.191.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:44167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.191.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
W20260806 02:40:50.840725 6477 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.191.62:35097,127.28.191.61:44167'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:40:50.841594 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:50.842592 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:50.851413 6486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.851452 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:50.851452 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:50.851903 6477 server_base.cc:1034] running on GCE node
I20260806 02:40:50.852504 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:50.853641 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:50.854812 6477 hybrid_clock.cc:648] HybridClock initialized: now 1785984050854800 us; error 24 us; skew 500 ppm
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:50.857582 6477 init.cc:377] Logged in from keytab as oryx/127.28.191.61@KRBTEST.COM (short username oryx)
I20260806 02:40:50.858480 6477 webserver.cc:460] Webserver started at http://127.28.191.61:40267/ using document root <none> and password file <none>
I20260806 02:40:50.859045 6477 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:50.859124 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:50.859362 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:50.861399 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/instance:
uuid: "07edf8dc8344473fa6768457546b2006"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "37e0789ea605b2cb87401177c92abf4d"
server_key_iv: "5e369ad1147f4713b154448a66f2ffac"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.861948 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal/instance:
uuid: "07edf8dc8344473fa6768457546b2006"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "37e0789ea605b2cb87401177c92abf4d"
server_key_iv: "5e369ad1147f4713b154448a66f2ffac"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.865530 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:40:50.867830 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:50.869032 6477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:50.869236 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "07edf8dc8344473fa6768457546b2006"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "37e0789ea605b2cb87401177c92abf4d"
server_key_iv: "5e369ad1147f4713b154448a66f2ffac"
server_key_version: "encryptionkey@0"
I20260806 02:40:50.869336 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:50.927012 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:50.930245 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:50.930456 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:50.937842 6477 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.61:44167
I20260806 02:40:50.938176 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.61:44167 every 8 connection(s)
I20260806 02:40:50.938848 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
I20260806 02:40:50.941829 6570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:50.945864 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 6477
I20260806 02:40:50.945828 6570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.61@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:50.959424 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.947948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.28.191.62:35097)
0806 02:40:50.948083 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.948087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.948356 (+ 269us) server_negotiation.cc:407] Connection header received
0806 02:40:50.948955 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.948958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.949014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.949151 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.950119 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.950753 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.951882 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.952170 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.952244 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.955086 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.955091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.955094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.955126 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.956849 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.957601 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.957607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.957610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.957763 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.958154 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.958157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.958158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.958456 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.958552 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.959117 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.959246 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:40:50.960336 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.947480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.28.191.61:44167)
0806 02:40:50.947889 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.947910 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.947950 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:40:50.948757 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.948776 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.948959 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.949219 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.950075 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.951142 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.951974 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.952452 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.952546 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.955572 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.955578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.955587 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.955607 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.957567 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.958095 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.958103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.958106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.958188 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.958470 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.958472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.958473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.958883 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.959099 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.959312 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.959427 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":280,"spinlock_wait_cycles":3712,"thread_start_us":99,"threads_started":1}
I20260806 02:40:50.960336 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.947866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52536)
0806 02:40:50.948232 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:50.948268 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:50.948767 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:50.949205 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:50.949211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:50.949711 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:50.949948 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.949962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.950905 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.950908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.951759 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.951768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.952298 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.952302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.952484 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.953289 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:50.953304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:50.954917 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:50.956967 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.956974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.956983 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.957458 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.957883 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.957891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.957893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.958046 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.958567 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:50.958573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:50.958883 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:40:50.959056 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:40:50.959377 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":44,"spinlock_wait_cycles":5376,"thread_start_us":137,"threads_started":1}
I20260806 02:40:50.962283 6570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:40:50 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984050, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.61@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:50.971035 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.962788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:36984)
0806 02:40:50.962877 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:50.962891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:50.962964 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:50.963170 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:50.963172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:50.963297 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:50.963387 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.963392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.964031 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.964034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.964528 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:50.964536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.964955 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.964958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:50.965025 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.965547 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:50.965566 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:50.967117 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:50.969297 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.969302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.969305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.969597 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.969870 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:50.969874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:50.969876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:50.969962 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:50.970463 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:50.970468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:50.970586 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 02:40:50.970711 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 02:40:50.970863 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 02:40:50.971697 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:50.962830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.28.191.61:44167)
0806 02:40:50.962941 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:50.962944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:50.962955 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:40:50.963037 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:50.963039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:50.963073 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:50.963137 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:50.963480 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.963944 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.964637 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:50.964866 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:50.964940 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:50.967256 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:50.967260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:50.967262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:50.967289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:50.969145 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.969693 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.969697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.969698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.969774 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:50.970062 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:50.970066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:50.970069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:50.970306 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:50.970412 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:50.971406 (+ 994us) server_negotiation.cc:299] Negotiation successful
0806 02:40:50.971529 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:40:50.975721 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Bootstrap starting.
I20260806 02:40:50.978713 6570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:50.979642 6570 log.cc:824] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:50.982082 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: No bootstrap required, opened a new log
I20260806 02:40:50.986804 6570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } }
I20260806 02:40:50.987095 6570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:50.987176 6570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07edf8dc8344473fa6768457546b2006, State: Initialized, Role: FOLLOWER
I20260806 02:40:50.987696 6570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } }
I20260806 02:40:50.988227 6580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } } }
I20260806 02:40:50.988415 6580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:50.989235 6570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:40:50.995319 6597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:40:50.995430 6597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:40:50.998935 6477 master_runner.cc:428] Detected that this master 07edf8dc8344473fa6768457546b2006 is joining an existing cluster
I20260806 02:40:50.999111 6477 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.191.61:44167
I20260806 02:40:51.010566 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.001727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.28.191.61:44167)
0806 02:40:51.001861 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.001866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.001882 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:40:51.001943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.001949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.001999 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.002081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.002692 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.003457 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.004702 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.005010 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.005113 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.006792 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.006796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.006799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.006821 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.008777 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.009294 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.009297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.009298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.009375 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.009647 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.009653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.009655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.009861 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.009969 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.010234 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.010373 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:40:51.011106 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.001287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:36990)
0806 02:40:51.001611 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.001627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.001790 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.002136 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.002140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.002409 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.002563 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.002572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.003602 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.003607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.004593 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.004604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.005276 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.005280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.005477 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.006261 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.006284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.006663 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.008898 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.008901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.008903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.009187 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.009474 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.009476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.009478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.009573 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.009960 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.009963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.010054 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.010775 (+ 721us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.010907 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":128,"spinlock_wait_cycles":2944,"thread_start_us":112,"threads_started":1}
I20260806 02:40:51.011335 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.001494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52548)
0806 02:40:51.001861 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.001876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.002027 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.002387 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.002388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.002561 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.002697 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.002704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.003753 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.003757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.004549 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.004559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.005276 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.005280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.005481 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.006861 (+ 1380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.006877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.007343 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.009627 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.009631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.009633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.009892 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.010224 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.010227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.010229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.010350 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.010819 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.010822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.010911 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.011025 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.011142 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":183,"spinlock_wait_cycles":1408,"thread_start_us":88,"threads_started":1}
I20260806 02:40:51.011391 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.001273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.28.191.62:35097)
0806 02:40:51.001410 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.001417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.001963 (+ 546us) server_negotiation.cc:407] Connection header received
0806 02:40:51.002141 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.002144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.002196 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.002245 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.002823 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.003602 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.004687 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.005037 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.005147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.007483 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.007488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.007491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.007521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.009505 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.010030 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.010033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.010035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.010124 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.010455 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.010458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.010460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.010682 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.010776 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.011109 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.011234 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.28.191.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.60@KRBTEST.COM" created.
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.62@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:51.033221 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.016739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.28.191.61:44167)
0806 02:40:51.016855 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.016859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.017462 (+ 603us) server_negotiation.cc:407] Connection header received
0806 02:40:51.017681 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.017684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.017732 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.017822 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.018667 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.019436 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.020856 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.021117 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.021213 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.029258 (+ 8045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.029263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.029266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.029292 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.031081 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.031689 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.031696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.031700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.031793 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.032123 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.032126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.032128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.032326 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.032443 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.032830 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.033000 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:40:51.033226 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.016733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:37004)
0806 02:40:51.017320 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.017373 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.017549 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.017857 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.017863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.018342 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.018524 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.018536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.019618 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.019624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.020702 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.020712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.021226 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.021230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.021342 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.025507 (+ 4165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.025520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.029047 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.031212 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.031225 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.031236 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.031567 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.031905 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.031909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.031911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.032012 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.032444 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.032455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.032596 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.032772 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.033005 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":87,"threads_started":1}
I20260806 02:40:51.034739 5851 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.191.61:44167: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "07edf8dc8344473fa6768457546b2006"
member_type: NON_VOTER
last_known_addr {
host: "127.28.191.61"
port: 44167
}
attrs {
promote: true
}
}
dest_uuid: "b10b88d1229a41088fd2232e3a46a931"
cas_config_opid_index: -1
I20260806 02:40:51.035494 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } }
I20260806 02:40:51.037520 6614 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 07edf8dc8344473fa6768457546b2006 (127.28.191.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } }
I20260806 02:40:51.038515 6529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:51.040330 6529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:40:51.040383 6614 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.191.61:44167
I20260806 02:40:51.040940 6477 master.cc:537] Master@127.28.191.61:44167 shutting down...
I20260806 02:40:51.041184 6617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:51.041951 6615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } }
I20260806 02:40:51.042050 6615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
W20260806 02:40:51.043277 5834 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 02:40:51.044489 5834 consensus_peers.cc:487] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 -> Peer 07edf8dc8344473fa6768457546b2006 (127.28.191.61:44167): Couldn't send request to peer 07edf8dc8344473fa6768457546b2006. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:40:51.046351 6477 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:51.046649 6477 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:51.046765 6477 tablet_replica.cc:330] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.60.keytab.
Entry for principal oryx/127.28.191.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.60.keytab.
I20260806 02:40:51.048439 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:35707
--webserver_interface=127.28.191.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.60.keytab
--principal=oryx/127.28.191.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:40:51.052593 6477 master.cc:552] Master@127.28.191.61:44167 shutdown complete.
I20260806 02:40:51.052649 6477 master_runner.cc:305] Clearing existing system tablet
I20260806 02:40:51.052901 6477 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.052963 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.054445 6477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:51.055423 6625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.055647 6477 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.055794 6477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:51.055837 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "07edf8dc8344473fa6768457546b2006"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "37e0789ea605b2cb87401177c92abf4d"
server_key_iv: "5e369ad1147f4713b154448a66f2ffac"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.058422 6477 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:51.061549 6477 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:40:51.061657 6477 log.cc:1196] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:40:51.061966 6477 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Deleting consensus metadata
I20260806 02:40:51.062141 6477 master_runner.cc:315] Copying system tablet from 127.28.191.62:35097
I20260806 02:40:51.063591 6477 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: tablet copy: Beginning tablet copy session from remote peer at address 127.28.191.62:35097
I20260806 02:40:51.073143 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.064298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.28.191.62:35097)
0806 02:40:51.064423 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.064428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.064644 (+ 216us) server_negotiation.cc:407] Connection header received
0806 02:40:51.064791 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.064794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.064841 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.064948 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.065548 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.066348 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.067541 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.067893 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.067994 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.069478 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.069482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.069485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.069519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.071253 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.071768 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.071772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.071774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.071859 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.072175 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.072178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.072180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.072380 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.072474 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.072811 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.072933 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:40:51.073167 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.064217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52564)
0806 02:40:51.064535 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.064553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.064679 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.065006 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.065010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.065280 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.065416 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.065424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.066471 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.066475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.067405 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.067417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.068024 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.068029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.068125 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.068769 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.068787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.069302 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.071376 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.071379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.071382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.071655 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.071961 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.071964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.071966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.072076 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.072479 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.072483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.072591 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.072751 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.072897 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260806 02:40:51.074358 5878 tablet_copy_service.cc:140] P b10b88d1229a41088fd2232e3a46a931: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 07edf8dc8344473fa6768457546b2006 ({username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:52564)
I20260806 02:40:51.074553 5878 tablet_copy_service.cc:161] P b10b88d1229a41088fd2232e3a46a931: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 07edf8dc8344473fa6768457546b2006 at {username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:52564: session id = 07edf8dc8344473fa6768457546b2006-00000000000000000000000000000000
I20260806 02:40:51.076390 5878 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:40:51.078348 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:40:51.080659 6477 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: tablet copy: Starting download of 0 data blocks...
I20260806 02:40:51.080847 6477 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: tablet copy: Starting download of 1 WAL segments...
I20260806 02:40:51.082615 6477 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:40:51.084479 5878 tablet_copy_service.cc:342] P b10b88d1229a41088fd2232e3a46a931: Request end of tablet copy session 07edf8dc8344473fa6768457546b2006-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:52564
I20260806 02:40:51.084594 5878 tablet_copy_service.cc:433] P b10b88d1229a41088fd2232e3a46a931: ending tablet copy session 07edf8dc8344473fa6768457546b2006-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 07edf8dc8344473fa6768457546b2006
I20260806 02:40:51.093129 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.094843 6636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.094940 6635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.098179 6477 server_base.cc:1034] running on GCE node
W20260806 02:40:51.098258 6639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.098475 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.098780 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.099923 6477 hybrid_clock.cc:648] HybridClock initialized: now 1785984051099911 us; error 34 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.101691 6477 init.cc:377] Logged in from keytab as oryx/127.28.191.61@KRBTEST.COM (short username oryx)
I20260806 02:40:51.102243 6477 webserver.cc:460] Webserver started at http://127.28.191.61:39325/ using document root <none> and password file <none>
I20260806 02:40:51.102447 6477 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.102496 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.103917 6477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:51.104972 6648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.105270 6477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:51.105356 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "07edf8dc8344473fa6768457546b2006"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "37e0789ea605b2cb87401177c92abf4d"
server_key_iv: "5e369ad1147f4713b154448a66f2ffac"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.105443 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.131255 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.133505 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.139365 6477 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.61:44167
I20260806 02:40:51.139422 6720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.61:44167 every 8 connection(s)
I20260806 02:40:51.140062 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
I20260806 02:40:51.142377 6721 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:40:51.143736 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Bootstrap starting.
I20260806 02:40:51.155185 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:40:51.155539 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Bootstrap complete.
I20260806 02:40:51.156160 6721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } }
I20260806 02:40:51.156298 6721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07edf8dc8344473fa6768457546b2006, State: Initialized, Role: LEARNER
I20260806 02:40:51.156414 6721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } }
I20260806 02:40:51.156804 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } }
I20260806 02:40:51.156915 6721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:51.157044 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:40:51.160564 6740 catalog_manager.cc:1259] Loaded cluster ID: 1d2d2c184bb645ec80bf9ebfef440b71
I20260806 02:40:51.160630 6740 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: loading cluster ID for follower catalog manager: success
I20260806 02:40:51.162889 6740 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: acquiring CA information for follower catalog manager: success
I20260806 02:40:51.164383 6740 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:40:51.182744 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:51.183036 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:51.183137 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:51.187044 6621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:51.187120 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:51.187158 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:51.187212 6621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:51.187260 6621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:51.192341 6621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.60.keytab
--principal=oryx/127.28.191.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.191.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:51.193414 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:51.194411 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.203080 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.203045 6751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.203115 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.203222 6621 server_base.cc:1034] running on GCE node
I20260806 02:40:51.203867 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.204946 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.206126 6621 hybrid_clock.cc:648] HybridClock initialized: now 1785984051206110 us; error 29 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.208722 6621 init.cc:377] Logged in from keytab as oryx/127.28.191.60@KRBTEST.COM (short username oryx)
I20260806 02:40:51.209614 6621 webserver.cc:460] Webserver started at http://127.28.191.60:45875/ using document root <none> and password file <none>
I20260806 02:40:51.210153 6621 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.210244 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.210469 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:51.212515 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/instance:
uuid: "59013bcdfcf2454a917f35d7e5c27900"
format_stamp: "Formatted at 2026-08-06 02:40:51 on dist-test-slave-tfpf"
server_key: "6e4a869cc8fe265f7b4f553ffdad4c27"
server_key_iv: "cc92b645f67ae53db2468d1b193fdbdb"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.213095 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal/instance:
uuid: "59013bcdfcf2454a917f35d7e5c27900"
format_stamp: "Formatted at 2026-08-06 02:40:51 on dist-test-slave-tfpf"
server_key: "6e4a869cc8fe265f7b4f553ffdad4c27"
server_key_iv: "cc92b645f67ae53db2468d1b193fdbdb"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.216589 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:40:51.218873 6764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.219996 6621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:51.220135 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "59013bcdfcf2454a917f35d7e5c27900"
format_stamp: "Formatted at 2026-08-06 02:40:51 on dist-test-slave-tfpf"
server_key: "6e4a869cc8fe265f7b4f553ffdad4c27"
server_key_iv: "cc92b645f67ae53db2468d1b193fdbdb"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.220237 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.230547 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.233321 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:51.233520 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.233775 6104 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
I20260806 02:40:51.240553 6621 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.60:35707
I20260806 02:40:51.240824 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.60:35707 every 8 connection(s)
I20260806 02:40:51.241580 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
I20260806 02:40:51.244346 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:51.246168 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 6621
I20260806 02:40:51.248759 6837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.60@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:51.260762 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.251115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.28.191.62:35097)
0806 02:40:51.251265 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.251270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.251523 (+ 253us) server_negotiation.cc:407] Connection header received
0806 02:40:51.251649 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.251652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.251700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.251783 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.252401 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.252921 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.253928 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.254246 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.254366 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.257125 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.257129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.257131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.257162 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.258821 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.259397 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.259403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.259406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.259491 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.259758 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.259761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.259762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.259982 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.260088 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.260468 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.260589 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:51.261576 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.247692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.28.191.60:35707)
0806 02:40:51.248130 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.248150 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.248198 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:40:51.249074 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.249123 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.249537 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.249895 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.250650 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.251955 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.252823 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.253377 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.253567 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.256319 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.256325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.256335 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.256357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.258435 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.258902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.258911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.258919 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.259034 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.259360 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.259366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.259368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.259894 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.260102 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.260344 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.260700 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260806 02:40:51.261574 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.250991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52580)
0806 02:40:51.251389 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.251436 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.251554 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.251837 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.251841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.252161 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.252274 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.252280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.253109 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.253112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.253743 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.253752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.254420 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.254425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.254530 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.255087 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.255103 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.256925 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.258929 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.258940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.258947 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.259298 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.259589 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.259592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.259593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.259667 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.260087 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.260093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.260240 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.260399 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.260785 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":126,"thread_start_us":147,"threads_started":1}
I20260806 02:40:51.262805 6837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.60@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:51.270816 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.263243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:37016)
0806 02:40:51.263630 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.263645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.263731 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.264000 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.264003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.264132 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.264217 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.264221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.265136 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.265140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.265646 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.265654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.266143 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.266145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.266214 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.266627 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.266638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.267799 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.269628 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.269630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.269632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.269824 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.270076 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.270078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.270079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.270149 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.270472 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.270474 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.270542 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.270621 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.270711 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260806 02:40:51.270992 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.263380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.28.191.61:44167)
0806 02:40:51.263640 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.263654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.263691 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:40:51.263837 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.263840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.263894 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.263963 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.264316 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.264966 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.265744 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.265984 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.266130 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.267935 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.267938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.267940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.267963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.269530 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.269907 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.269910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.269911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.269985 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.270225 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.270227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.270229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.270393 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.270468 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.270678 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.270778 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":119,"threads_started":1}
I20260806 02:40:51.271731 6837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.60@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
I20260806 02:40:51.279378 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.272223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49360)
0806 02:40:51.272356 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.272371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.272441 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.272674 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.272676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.272795 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.272899 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.272905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.274200 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.274203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.274671 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.274679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.275087 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.275090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.275143 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.275500 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.275511 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.276571 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.278199 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.278202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.278203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.278375 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.278617 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.278619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.278621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.278690 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.278992 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.278994 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.279069 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.279146 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.279230 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:40:51.279397 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.272296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.28.191.60:35707)
0806 02:40:51.272397 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.272400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.272411 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:40:51.272544 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.272546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.272579 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.272643 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.272999 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.273627 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.274772 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.274986 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.275062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.276688 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.276690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.276692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.276713 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.278106 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.278455 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.278457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.278458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.278538 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.278764 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.278766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.278767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.278914 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.278987 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.279201 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.279286 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:40:51.282446 6837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Bootstrap starting.
I20260806 02:40:51.284727 6837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:51.285533 6837 log.cc:824] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:51.287396 6837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: No bootstrap required, opened a new log
I20260806 02:40:51.290405 6837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } }
I20260806 02:40:51.290596 6837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:51.290680 6837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59013bcdfcf2454a917f35d7e5c27900, State: Initialized, Role: FOLLOWER
I20260806 02:40:51.291100 6837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } }
I20260806 02:40:51.291600 6844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } } }
I20260806 02:40:51.291733 6844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:51.292309 6837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:40:51.297292 6858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:40:51.297384 6858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:40:51.300563 6621 master_runner.cc:428] Detected that this master 59013bcdfcf2454a917f35d7e5c27900 is joining an existing cluster
I20260806 02:40:51.300627 6621 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.191.60:35707
I20260806 02:40:51.312233 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.303835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.28.191.61:44167)
0806 02:40:51.304118 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.304122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.304134 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:51.304169 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.304175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.304226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.304293 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.305005 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305771 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307876 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.308100 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.308205 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.309297 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.309301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.309304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.309333 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.310668 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.311089 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.311093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.311094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.311174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.311446 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.311451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.311452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.311624 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.311729 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.311983 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.312104 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260806 02:40:51.312247 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.302999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:37030)
0806 02:40:51.303325 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.303340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.303436 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.304405 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.304409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.304647 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.304840 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.304847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305887 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.305889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.307758 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.307765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.308216 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.308220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.308309 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.308736 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.308747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.309172 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.310777 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.310781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.310783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.310976 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.311265 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.311269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.311271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.311359 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.311720 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.311723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.311826 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.311931 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.312069 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":63,"thread_start_us":114,"threads_started":1}
I20260806 02:40:51.314895 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5930
I20260806 02:40:51.331368 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.303053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.28.191.62:35097)
0806 02:40:51.303396 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.303400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.303714 (+ 314us) server_negotiation.cc:407] Connection header received
0806 02:40:51.303877 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.303880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.303930 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.303984 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.304748 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305606 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.306926 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.307248 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307324 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.327213 (+ 19889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.327218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.327221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.327253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.329424 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.330036 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.330041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.330042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.330128 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.330441 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.330444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.330446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.330672 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.330741 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.331088 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.331210 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260806 02:40:51.331434 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.303945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.28.191.60:35707)
0806 02:40:51.304055 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.304059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.304071 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:51.304102 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.304104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.304146 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.304217 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.304958 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305722 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307029 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.307363 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.327439 (+ 19966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.327446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.327451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.327479 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.329387 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.329959 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.329965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.329969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.330057 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.330333 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.330338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.330343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.330584 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.330668 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.331065 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.331199 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":44}
I20260806 02:40:51.331897 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.302999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52582)
0806 02:40:51.303637 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.303649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.303773 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.304138 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.304141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.304438 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.304597 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.304606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305781 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.305784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.306751 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.306762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.307491 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.307626 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.308200 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.308215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.327001 (+ 18786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.329593 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.329596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.329598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.329916 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.330239 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.330242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.330244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.330326 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.330813 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.330816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.330904 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.331584 (+ 680us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.331729 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":91,"thread_start_us":83,"threads_started":1}
I20260806 02:40:51.332733 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.303450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49370)
0806 02:40:51.303732 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.303746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.303839 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.304324 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.304327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.304622 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.304839 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.304847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.305881 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.305884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.306914 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.306925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.307480 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.307483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.307675 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.326710 (+ 19035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.326733 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.327285 (+ 552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.329510 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.329516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.329520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.329859 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.330156 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.330161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.330166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.330249 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.330732 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.330738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.330851 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.332314 (+ 1463us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.332471 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":95,"spinlock_wait_cycles":4864,"thread_start_us":120,"threads_started":1}
I20260806 02:40:51.334401 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:42503
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:35097
--webserver_port=32889
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984049, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.62@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
I20260806 02:40:51.345860 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.335886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49384)
0806 02:40:51.336106 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.336119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.336190 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.336554 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.336557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.336753 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.336865 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.336873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.337809 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.337813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.338645 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.338656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.339298 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.339304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.339417 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.339990 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.340005 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.341845 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.343858 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.343864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.343868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.344303 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.344679 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.344683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.344685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.344788 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.345318 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.345321 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.345427 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.345550 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.345693 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 02:40:51.345975 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.336124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.28.191.60:35707)
0806 02:40:51.336265 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.336271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.336284 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:51.336358 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.336363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.336423 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.336513 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.336999 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.337704 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.338775 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.339173 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.339297 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.341994 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.341998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.342001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.342026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.343735 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.344460 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.344466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.344470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.344565 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.344907 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.344912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.344914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.345204 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.345323 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.345629 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.345765 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:40:51.347463 5851 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.191.60:35707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900"
member_type: NON_VOTER
last_known_addr {
host: "127.28.191.60"
port: 35707
}
attrs {
promote: true
}
}
dest_uuid: "b10b88d1229a41088fd2232e3a46a931"
cas_config_opid_index: 5
I20260806 02:40:51.348130 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } }
I20260806 02:40:51.349257 6617 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.350797 6617 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.191.60:35707
I20260806 02:40:51.351013 6618 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.351128 6618 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:51.351660 6621 master.cc:537] Master@127.28.191.60:35707 shutting down...
I20260806 02:40:51.352180 6803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:51.353724 6803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:40:51.355060 6614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:40:51.356820 5834 consensus_peers.cc:487] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 -> Peer 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707): Couldn't send request to peer 59013bcdfcf2454a917f35d7e5c27900. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:40:51.358394 6621 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:51.358744 6621 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:51.358904 6621 tablet_replica.cc:330] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: stopping tablet replica
I20260806 02:40:51.362656 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.351488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.28.191.61:44167)
0806 02:40:51.351606 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.351610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.357443 (+ 5833us) server_negotiation.cc:407] Connection header received
0806 02:40:51.357556 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.357562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.357666 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.357763 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:51.358257 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.359078 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.361053 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.361732 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.361812 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.361958 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.362444 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.362518 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:40:51.363027 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.351229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:37036)
0806 02:40:51.357350 (+ 6121us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.357367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.357464 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.357953 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.357957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.357977 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:51.358120 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.358129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.359624 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.359627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.360906 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.360917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.362179 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.362183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.362279 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.362328 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.362838 (+ 510us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.362916 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6039}
I20260806 02:40:51.363906 6684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 LEARNER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:40:51.364398 6614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:40:51.366304 6684 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.368607 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b10b88d1229a41088fd2232e3a46a931. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.368716 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:40:51.368801 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: NON_VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: true } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.368862 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:40:51.369568 6614 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: attempting to promote NON_VOTER 07edf8dc8344473fa6768457546b2006 to VOTER
I20260806 02:40:51.369972 6614 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } }
I20260806 02:40:51.371167 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.370868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49388)
0806 02:40:51.370998 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.371097 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:40:51.371268 6684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 LEARNER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 02:40:51.371757 6617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:40:51.372257 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.371950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49400)
0806 02:40:51.372151 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.372199 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260806 02:40:51.372370 5834 consensus_peers.cc:487] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 -> Peer 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707): Couldn't send request to peer 59013bcdfcf2454a917f35d7e5c27900. Status: Network error: Client connection negotiation failed: client connection to 127.28.191.60:35707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:40:51.373322 6618 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 07edf8dc8344473fa6768457546b2006 (127.28.191.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.374315 6621 master.cc:552] Master@127.28.191.60:35707 shutdown complete.
I20260806 02:40:51.374383 6621 master_runner.cc:305] Clearing existing system tablet
I20260806 02:40:51.374619 6621 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.374691 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.374807 6618 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.374913 6618 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:51.375039 6618 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.375124 6618 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:51.375658 6684 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 07edf8dc8344473fa6768457546b2006 (127.28.191.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.376147 6621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 02:40:51.377020 6873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.377234 6621 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.377345 6621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:51.377385 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "59013bcdfcf2454a917f35d7e5c27900"
format_stamp: "Formatted at 2026-08-06 02:40:51 on dist-test-slave-tfpf"
server_key: "6e4a869cc8fe265f7b4f553ffdad4c27"
server_key_iv: "cc92b645f67ae53db2468d1b193fdbdb"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.377506 6871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.379359 6621 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:51.381155 6871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:51.381681 6621 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:40:51.381772 6621 log.cc:1196] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:40:51.382019 6621 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Deleting consensus metadata
I20260806 02:40:51.382186 6621 master_runner.cc:315] Copying system tablet from 127.28.191.62:35097
I20260806 02:40:51.383299 6621 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: tablet copy: Beginning tablet copy session from remote peer at address 127.28.191.62:35097
I20260806 02:40:51.392387 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.383797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:52590)
0806 02:40:51.384107 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.384121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.384229 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.384597 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.384600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.384825 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.384956 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.384964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.386015 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.386019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.386962 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.386972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.387554 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.387558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.387673 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.388255 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.388269 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.388687 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.390735 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.390738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.390740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.390988 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.391281 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.391284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.391286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.391375 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.391809 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.391812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.391907 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.392012 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.392138 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":69,"threads_started":1}
I20260806 02:40:51.392643 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.383870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.28.191.62:35097)
0806 02:40:51.383975 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.383980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.384225 (+ 245us) server_negotiation.cc:407] Connection header received
0806 02:40:51.384345 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.384349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.384395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.384443 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.385108 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.385895 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.387091 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.387426 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.387538 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.388831 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.388835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.388837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.388862 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.390620 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.391093 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.391096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.391098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.391185 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.391478 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.391483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.391485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.391701 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.391798 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.392119 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.392227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:40:51.393216 5878 tablet_copy_service.cc:140] P b10b88d1229a41088fd2232e3a46a931: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 59013bcdfcf2454a917f35d7e5c27900 ({username='oryx', principal='oryx/127.28.191.60@KRBTEST.COM'} at 127.0.0.1:52590)
I20260806 02:40:51.393325 5878 tablet_copy_service.cc:161] P b10b88d1229a41088fd2232e3a46a931: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 59013bcdfcf2454a917f35d7e5c27900 at {username='oryx', principal='oryx/127.28.191.60@KRBTEST.COM'} at 127.0.0.1:52590: session id = 59013bcdfcf2454a917f35d7e5c27900-00000000000000000000000000000000
I20260806 02:40:51.394387 5878 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:40:51.395941 6621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:40:51.398224 6621 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: tablet copy: Starting download of 0 data blocks...
I20260806 02:40:51.398403 6621 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: tablet copy: Starting download of 1 WAL segments...
I20260806 02:40:51.399806 6621 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:40:51.401472 5878 tablet_copy_service.cc:342] P b10b88d1229a41088fd2232e3a46a931: Request end of tablet copy session 59013bcdfcf2454a917f35d7e5c27900-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.191.60@KRBTEST.COM'} at 127.0.0.1:52590
I20260806 02:40:51.401558 5878 tablet_copy_service.cc:433] P b10b88d1229a41088fd2232e3a46a931: ending tablet copy session 59013bcdfcf2454a917f35d7e5c27900-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 59013bcdfcf2454a917f35d7e5c27900
I20260806 02:40:51.408185 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.409817 6881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.409799 6882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.414140 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.414314 6621 server_base.cc:1034] running on GCE node
I20260806 02:40:51.414466 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.414764 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.415900 6621 hybrid_clock.cc:648] HybridClock initialized: now 1785984051415885 us; error 31 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.417603 6621 init.cc:377] Logged in from keytab as oryx/127.28.191.60@KRBTEST.COM (short username oryx)
I20260806 02:40:51.418059 6621 webserver.cc:460] Webserver started at http://127.28.191.60:35731/ using document root <none> and password file <none>
I20260806 02:40:51.418246 6621 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.418298 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.419579 6621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:51.420677 6896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.420921 6621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:51.421006 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "59013bcdfcf2454a917f35d7e5c27900"
format_stamp: "Formatted at 2026-08-06 02:40:51 on dist-test-slave-tfpf"
server_key: "6e4a869cc8fe265f7b4f553ffdad4c27"
server_key_iv: "cc92b645f67ae53db2468d1b193fdbdb"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.421161 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.428260 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.430531 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.435600 6621 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.60:35707
I20260806 02:40:51.435765 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.60:35707 every 8 connection(s)
I20260806 02:40:51.436254 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
I20260806 02:40:51.438215 6971 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:40:51.439286 6971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Bootstrap starting.
I20260806 02:40:51.444965 6282 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
I20260806 02:40:51.451380 6971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:40:51.451725 6971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: Bootstrap complete.
I20260806 02:40:51.452306 6971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } }
I20260806 02:40:51.452457 6971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 59013bcdfcf2454a917f35d7e5c27900, State: Initialized, Role: LEARNER
I20260806 02:40:51.452596 6971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } }
I20260806 02:40:51.452970 6974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.453138 6971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:51.453240 6974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:40:51.457685 6987 catalog_manager.cc:1259] Loaded cluster ID: 1d2d2c184bb645ec80bf9ebfef440b71
I20260806 02:40:51.457742 6987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: loading cluster ID for follower catalog manager: success
I20260806 02:40:51.459944 6987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: acquiring CA information for follower catalog manager: success
I20260806 02:40:51.461359 6987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:40:51.480757 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:51.481027 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:51.481127 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:51.485145 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:51.485215 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:51.485298 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:51.490370 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:42503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=32889
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:51.491619 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:51.492933 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.503579 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.503593 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.503634 6995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.504385 6869 server_base.cc:1034] running on GCE node
I20260806 02:40:51.504910 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.505774 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.506976 6869 hybrid_clock.cc:648] HybridClock initialized: now 1785984051506942 us; error 50 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.511513 6869 init.cc:377] Logged in from keytab as oryx/127.28.191.1@KRBTEST.COM (short username oryx)
I20260806 02:40:51.512346 6869 webserver.cc:460] Webserver started at http://127.28.191.1:32889/ using document root <none> and password file <none>
I20260806 02:40:51.512897 6869 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.513011 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.517760 6869 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:51.520686 7004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.522713 6869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:40:51.522804 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7975b79b3b5d4bf7b27b0c11fedef25e"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c5def424cd6a5113c67901c983b28e35"
server_key_iv: "0ed2995a1767cd355dc5092c5e9dd0b1"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.523320 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.545920 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.550205 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:51.550411 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.551045 6869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:51.551854 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:51.551911 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.551946 6869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:51.551970 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.561197 6869 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42503
I20260806 02:40:51.561267 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42503 every 8 connection(s)
I20260806 02:40:51.562184 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:51.562832 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 6869
I20260806 02:40:51.563030 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 6111
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:51.583153 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.565451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:48615 (local address 127.28.191.62:35097)
0806 02:40:51.565569 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.565575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.566790 (+ 1215us) server_negotiation.cc:407] Connection header received
0806 02:40:51.567571 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.567578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.567647 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.567709 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.569751 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.570935 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.573485 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.574012 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.574121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.578403 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.578406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.578409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.578453 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.580308 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.580985 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.580991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.580994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.581108 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.581565 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.581570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.581575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.581838 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.582237 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.582853 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.582979 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:51.584972 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:42979
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:35097
--webserver_port=41481
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
I20260806 02:40:51.585672 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.564373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:48615)
0806 02:40:51.566673 (+ 2300us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.566694 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.567312 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.568071 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.568081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.568552 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.569523 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.569555 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.571161 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.571168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.573292 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.573308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.574385 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.574395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.574704 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.576098 (+ 1394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.576119 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.578201 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.580446 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.580451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.580465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.580819 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.581246 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.581251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.581254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.581438 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.582311 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.582317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.582588 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.583406 (+ 818us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.583752 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"mutex_wait_us":1196,"thread_start_us":849,"threads_started":1}
I20260806 02:40:51.587565 7167 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:51.587798 7167 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.588045 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.575760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:43375 (local address 127.28.191.61:44167)
0806 02:40:51.575998 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.576004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.577136 (+ 1132us) server_negotiation.cc:407] Connection header received
0806 02:40:51.577321 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.577327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.577393 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.577451 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.578161 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.579075 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.580408 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.580801 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.580921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.584105 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.584111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.584117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.584149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.586043 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.586712 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.586715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.586717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.586813 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.587140 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.587143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.587145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.587345 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.587420 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.587788 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.587911 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 02:40:51.588349 7167 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
I20260806 02:40:51.589197 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.575659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.28.191.1:43375)
0806 02:40:51.577028 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.577043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.577211 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.577612 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.577617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.577851 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.578022 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.578032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.579240 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.579246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.580263 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.580278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.581295 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.581314 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.581561 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.582260 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.582278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.583931 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.586330 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.586334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.586337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.586578 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.586935 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.586939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.586942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.587035 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.587490 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.587493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.587584 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.588217 (+ 633us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.588343 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":96,"threads_started":1}
I20260806 02:40:51.589673 5851 ts_manager.cc:194] Re-registered known tserver with Master: 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503)
I20260806 02:40:51.589969 7168 heartbeater.cc:347] Connected to a master server at 127.28.191.61:44167
I20260806 02:40:51.590058 7168 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.590225 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.564371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.28.191.1:60099)
0806 02:40:51.565544 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.565599 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.567289 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.578653 (+ 11364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.578658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.578848 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.579043 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.579055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.580581 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.580584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.582781 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.582792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.583801 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.583805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.583892 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.584341 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.584355 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.585838 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.587740 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.587746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.587750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.588034 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.588626 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.588631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.588635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.588747 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.589446 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.589452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.589558 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.589663 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.589818 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":775,"threads_started":1}
I20260806 02:40:51.590265 7168 heartbeater.cc:511] Master 127.28.191.61:44167 requested a full tablet report, sending...
I20260806 02:40:51.590409 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:48615
I20260806 02:40:51.590827 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.577422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:60099 (local address 127.28.191.60:35707)
0806 02:40:51.578358 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.578373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.578385 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:51.578442 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.578445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.578493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.579172 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.579309 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.580460 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.582917 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.583663 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.584104 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.585988 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.585995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.586001 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.586031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.587549 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.588449 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.588452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.588454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.588535 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.588863 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.588868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.588869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.589190 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.590430 (+ 1240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.590523 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.590662 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":82,"threads_started":1}
I20260806 02:40:51.591568 7162 heartbeater.cc:347] Connected to a master server at 127.28.191.60:35707
I20260806 02:40:51.591717 7162 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.591888 7162 heartbeater.cc:511] Master 127.28.191.60:35707 requested a full tablet report, sending...
I20260806 02:40:51.591876 6669 ts_manager.cc:194] Registered new tserver with Master: 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503)
I20260806 02:40:51.592902 6921 ts_manager.cc:194] Registered new tserver with Master: 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503)
I20260806 02:40:51.681324 6472 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
W20260806 02:40:51.702080 7175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:51.702297 7175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:51.702343 7175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:51.706276 7175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:51.706331 7175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:51.706386 7175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:51.711340 7175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:42979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=41481
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:51.712226 7175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:51.713150 7175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.723708 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.723721 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.723722 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.724093 7175 server_base.cc:1034] running on GCE node
I20260806 02:40:51.724501 7175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.725116 7175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.726271 7175 hybrid_clock.cc:648] HybridClock initialized: now 1785984051726245 us; error 45 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.728906 7175 init.cc:377] Logged in from keytab as oryx/127.28.191.2@KRBTEST.COM (short username oryx)
I20260806 02:40:51.729777 7175 webserver.cc:460] Webserver started at http://127.28.191.2:41481/ using document root <none> and password file <none>
I20260806 02:40:51.730322 7175 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.730435 7175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.734162 7175 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:51.736152 7194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.737390 7175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:51.737533 7175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "bc2a8490f83a4a3889ae7e5193e23922"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "c65aac5343631d37faf9346e1444930f"
server_key_iv: "78619823967dd166352151317054b84a"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.738076 7175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.760003 7175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.763046 7175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:51.763200 7175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.763707 7175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:51.764504 7175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:51.764560 7175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.764598 7175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:51.764613 7175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.773525 7175 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:42979
I20260806 02:40:51.773586 7349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:42979 every 8 connection(s)
I20260806 02:40:51.774542 7175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:51.783957 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 7175
I20260806 02:40:51.784164 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 6290
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
I20260806 02:40:51.791642 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.776293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:51945 (local address 127.28.191.61:44167)
0806 02:40:51.776445 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.776449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.777422 (+ 973us) server_negotiation.cc:407] Connection header received
0806 02:40:51.778486 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.778490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.778561 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.778639 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.780006 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780790 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782173 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.782539 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782619 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.787530 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.787534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.787537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.787571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.789160 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.789824 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.789828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.789832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.789919 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.790332 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.790336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.790337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.790531 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.790598 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.791333 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.791459 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:40:51.792241 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.778149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:37737 (local address 127.28.191.60:35707)
0806 02:40:51.778307 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.778311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.778327 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:40:51.778641 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.778644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.778694 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.778768 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.780006 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780791 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782173 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.782565 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782634 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.786891 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.786896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.786899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.786927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.789374 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.789900 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.789908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.789912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.790030 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.790325 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.790332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.790337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.790670 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.790822 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.791350 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.791549 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:51.792670 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.777727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.28.191.2:37737)
0806 02:40:51.777982 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.777996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.778537 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.778818 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.778827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.779323 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.779859 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.779868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780937 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.780940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.782028 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.782039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.782795 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.782988 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.783620 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.783656 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.786722 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.789520 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.789524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.789526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.789790 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.790144 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.790147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.790149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.790221 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.790813 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.790815 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.791024 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.791264 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.791611 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":227,"spinlock_wait_cycles":74496,"thread_start_us":104,"threads_started":1}
I20260806 02:40:51.792954 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.776712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.28.191.2:51945)
0806 02:40:51.777270 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.777320 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.778259 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.778818 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.778827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.779323 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.779859 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.779867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780937 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.780940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.782028 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.782046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.782776 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.782992 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.783695 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.783709 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.787394 (+ 3685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.789273 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.789279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.789293 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.789698 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.790017 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.790019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.790021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.790215 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.790659 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.790664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.791024 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.791791 (+ 767us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.791916 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":309,"spinlock_wait_cycles":29824,"thread_start_us":130,"threads_started":1}
I20260806 02:40:51.794225 7351 heartbeater.cc:347] Connected to a master server at 127.28.191.60:35707
I20260806 02:40:51.794463 7351 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.795096 7351 heartbeater.cc:511] Master 127.28.191.60:35707 requested a full tablet report, sending...
I20260806 02:40:51.796955 6921 ts_manager.cc:194] Registered new tserver with Master: bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:51.802134 7354 heartbeater.cc:347] Connected to a master server at 127.28.191.61:44167
I20260806 02:40:51.802237 7354 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.802438 7354 heartbeater.cc:511] Master 127.28.191.61:44167 requested a full tablet report, sending...
I20260806 02:40:51.803188 6669 ts_manager.cc:194] Registered new tserver with Master: bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:51.805817 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.776286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:40327 (local address 127.28.191.62:35097)
0806 02:40:51.776441 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.776446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.777549 (+ 1103us) server_negotiation.cc:407] Connection header received
0806 02:40:51.778468 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.778471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.778537 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.778636 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.779985 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780511 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.781764 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.781997 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782113 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.802103 (+ 19990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:51.802110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:51.802113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:51.802149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:51.804006 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.804471 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.804475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.804478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.804567 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:51.804856 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:51.804860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:51.804862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:51.805167 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:51.805274 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.805551 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.805667 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:40:51.805843 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.776712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:40327)
0806 02:40:51.777451 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.777466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.778262 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.778818 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.778827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.779313 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.779846 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.779858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.780651 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.780654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.781628 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.781639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.782162 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.782165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.782355 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.782967 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:51.782984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:51.785790 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:51.804134 (+ 18344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.804139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.804141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.804362 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.804665 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:51.804670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:51.804672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:51.804760 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:51.805264 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:51.805267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:51.805371 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.805482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.805631 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":30,"spinlock_wait_cycles":768,"thread_start_us":139,"threads_started":1}
I20260806 02:40:51.806710 7353 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:51.806806 7353 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:51.807003 7353 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
I20260806 02:40:51.807680 5851 ts_manager.cc:194] Re-registered known tserver with Master: bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:51.808247 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:40327
I20260806 02:40:51.809391 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:38491
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:35097
--webserver_port=33897
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:51.923257 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:51.923485 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:51.923581 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:51.927639 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:51.927688 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:51.927793 7362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:51.932693 7362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:38491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=33897
--tserver_master_addrs=127.28.191.62:35097,127.28.191.61:44167,127.28.191.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:51.933681 7362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:51.934646 7362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:51.943923 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:51.943949 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.944077 7362 server_base.cc:1034] running on GCE node
W20260806 02:40:51.943933 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:51.944629 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:51.945263 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:51.946448 7362 hybrid_clock.cc:648] HybridClock initialized: now 1785984051946397 us; error 63 us; skew 500 ppm
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:51.949601 7362 init.cc:377] Logged in from keytab as oryx/127.28.191.3@KRBTEST.COM (short username oryx)
I20260806 02:40:51.950429 7362 webserver.cc:460] Webserver started at http://127.28.191.3:33897/ using document root <none> and password file <none>
I20260806 02:40:51.951032 7362 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:51.951145 7362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:51.952684 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.948993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.28.191.60:35707)
0806 02:40:51.949168 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.949174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.949377 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:40:51.949592 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.949596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.949653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.949756 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:51.950057 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.950654 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.951725 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.952147 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.952269 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.952414 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:51.952493 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 02:40:51.952560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:40:51.952729 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.948904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49412)
0806 02:40:51.949273 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.949287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.949467 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.949781 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.949784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.949804 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:51.949918 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.949925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.950774 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.950778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.951605 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.951615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.952285 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.952288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:51.952373 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:51.952397 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:40:51.952480 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:40:51.952560 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260806 02:40:51.953486 6932 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:40:51.954025 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b10b88d1229a41088fd2232e3a46a931. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:51.954226 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:40:51.955546 7362 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:40:51.957563 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.958889 7362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:51.959000 7362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "25afa3a9cdf343b3977affe8f30b3ee8"
format_stamp: "Formatted at 2026-08-06 02:40:50 on dist-test-slave-tfpf"
server_key: "35ab5e8e3c9c0c76482ccbf801027f74"
server_key_iv: "0ab458e234913a4fc61195c74e3b8b50"
server_key_version: "encryptionkey@0"
I20260806 02:40:51.959518 7362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:51.976894 7362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:51.979810 7362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:51.980011 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:51.980583 7362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:40:51.981436 7362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:51.981510 7362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.981575 7362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:51.981626 7362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:51.990871 7362 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:38491
I20260806 02:40:51.990903 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:38491 every 8 connection(s)
I20260806 02:40:51.991856 7362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:51.996062 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 7362
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
Aug 06 02:40:51 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.61@KRBTEST.COM
Aug 06 02:40:52 dist-test-slave-tfpf krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984051, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.60@KRBTEST.COM
I20260806 02:40:52.010696 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.993761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:54065 (local address 127.28.191.62:35097)
0806 02:40:51.993908 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.993913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.994964 (+ 1051us) server_negotiation.cc:407] Connection header received
0806 02:40:51.996211 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.996214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.996272 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.996328 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.998674 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.999422 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.002050 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.002385 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.002457 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.005361 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.005368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.005374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.005406 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.007134 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.007863 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.007868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.007873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.008003 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.008452 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.008456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.008457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.008702 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.008792 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.009572 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.010531 (+ 959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:40:52.011549 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.994151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:54065)
0806 02:40:51.994863 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.994877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.995960 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.996753 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.996762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.997586 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.998539 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.998545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.001048 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.001053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.001920 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.001931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.002621 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.002624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.002720 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.003168 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:52.003194 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:52.005042 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:52.007264 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.007269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.007284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.007711 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.008124 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.008129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.008133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.008304 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.008911 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:52.008920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:52.009286 (+ 366us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.009718 (+ 432us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.010071 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":118,"spinlock_wait_cycles":19584,"thread_start_us":148,"threads_started":1}
I20260806 02:40:52.012655 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.995146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.28.191.3:55663)
0806 02:40:51.995428 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.995443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.995965 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.996754 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.996763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.997474 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.998290 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.998313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.999568 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.999572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.001171 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.001185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.001969 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.001975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.002216 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.003363 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:52.003377 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:52.005565 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:52.010887 (+ 5322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.010890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.010892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.011158 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.011488 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.011492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.011495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.011586 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.012051 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:52.012056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:52.012149 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.012284 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.012435 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":158,"thread_start_us":94,"threads_started":1}
I20260806 02:40:52.013235 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.995383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:55663 (local address 127.28.191.61:44167)
0806 02:40:51.995492 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.995496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.995509 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:51.996258 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.996261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.996315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.996388 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.998463 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.999307 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.001320 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.001687 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.001797 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.005708 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.005712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.005716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.005743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.009690 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.011278 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.011283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.011287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.011375 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.011698 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.011701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.011703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.011909 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.011992 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.012920 (+ 928us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.013083 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:40:52.013293 7565 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35097
I20260806 02:40:52.013557 7565 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:52.013826 7566 heartbeater.cc:347] Connected to a master server at 127.28.191.61:44167
I20260806 02:40:52.013916 7566 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:52.014161 7565 heartbeater.cc:511] Master 127.28.191.62:35097 requested a full tablet report, sending...
I20260806 02:40:52.014535 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.998272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.28.191.61:44167)
0806 02:40:51.999526 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.999532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.999898 (+ 366us) server_negotiation.cc:407] Connection header received
0806 02:40:51.999937 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.999940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.999985 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.000305 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.001326 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.002090 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.003259 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.003626 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.003723 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.006824 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.006830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.006835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.006866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.012042 (+ 5176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.012570 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.012574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.012576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.012667 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.012960 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.012963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.012964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.013207 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.013341 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.014272 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.014389 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":126,"threads_started":1}
I20260806 02:40:52.014163 7566 heartbeater.cc:511] Master 127.28.191.61:44167 requested a full tablet report, sending...
I20260806 02:40:52.015982 6669 ts_manager.cc:194] Registered new tserver with Master: 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:52.016062 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.993820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:54229 (local address 127.28.191.60:35707)
0806 02:40:51.993940 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.993944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:51.994852 (+ 908us) server_negotiation.cc:407] Connection header received
0806 02:40:51.996240 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:51.996243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:51.996302 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:51.996354 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:51.998958 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.999726 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.002733 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.003125 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.003218 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.009160 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.009163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.009166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.009192 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.011267 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.014722 (+ 3455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.014726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.014727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.014814 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.015166 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.015170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.015172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.015374 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.015457 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.015799 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.015914 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260806 02:40:52.016136 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.999112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.28.191.62:35097)
0806 02:40:51.999612 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:51.999616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.000525 (+ 909us) server_negotiation.cc:407] Connection header received
0806 02:40:52.000686 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.000688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.000731 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.000790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.003261 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.004091 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.005247 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.006144 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.006236 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.008263 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.008267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.008270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.008299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.010406 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.014742 (+ 4336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.014745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.014747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.014840 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.015176 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.015181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.015183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.015380 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.015492 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.015833 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.015961 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":68,"threads_started":1}
I20260806 02:40:52.016896 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.994151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.28.191.3:54229)
0806 02:40:51.994694 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:51.994745 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:51.995960 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:51.996753 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:51.996762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:51.997486 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:51.998507 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:51.998518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:51.999875 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:51.999880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.002599 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.002610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.005049 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.005053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.006913 (+ 1860us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.007372 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:52.007387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:52.008964 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:52.013868 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.013871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.013873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.014119 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.014946 (+ 827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:52.014950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:52.014953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:52.015043 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:52.015515 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:52.015521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:52.015616 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.016200 (+ 584us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.016356 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":168,"spinlock_wait_cycles":12544,"thread_start_us":104,"threads_started":1}
I20260806 02:40:52.017556 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:51.998751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.28.191.60:35707)
0806 02:40:52.000140 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.000144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.000157 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:52.000191 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.000194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.000236 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.003128 (+ 2892us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.004073 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.004859 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.010079 (+ 5220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.010447 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.010878 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.011989 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.011992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.011994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.012020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.014550 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.015778 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.015782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.015783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.015867 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.016248 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.016251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.016252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.016442 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.017190 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.017269 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.017398 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":671,"server-negotiator.queue_time_us":1318,"thread_start_us":62,"threads_started":1}
I20260806 02:40:52.017736 5851 ts_manager.cc:194] Re-registered known tserver with Master: 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:52.017822 7562 heartbeater.cc:347] Connected to a master server at 127.28.191.60:35707
I20260806 02:40:52.017897 7562 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:52.018306 7562 heartbeater.cc:511] Master 127.28.191.60:35707 requested a full tablet report, sending...
I20260806 02:40:52.018457 5851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:54065
I20260806 02:40:52.019165 6921 ts_manager.cc:194] Registered new tserver with Master: 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:52.019572 5851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:52.022794 5851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:52.025300 7589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: NON_VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: true } } }
I20260806 02:40:52.025460 7589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:52.028553 7589 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: attempting to promote NON_VOTER 59013bcdfcf2454a917f35d7e5c27900 to VOTER
I20260806 02:40:52.028954 7589 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:52.031766 6932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 LEARNER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:40:52.031904 6684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Refusing update from remote peer b10b88d1229a41088fd2232e3a46a931: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:40:52.032164 7364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:40:52.032572 7364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:40:52.034438 7588 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.034829 6932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.036013 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.036118 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:52.036242 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.036353 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:52.036475 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.036577 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:52.036655 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.036806 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:52.036675 6684 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.038169 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b10b88d1229a41088fd2232e3a46a931" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:52.038300 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:52.044296 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.039004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:38491 (local address 127.0.0.1:43486)
0806 02:40:52.039121 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:52.039135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:52.039214 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:52.040089 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:52.040091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:52.040106 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:52.040208 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.040215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.041545 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.041549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.042697 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.042706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.043931 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.043934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.044026 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.044053 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.044103 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.044171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:40:52.045017 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.039212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.28.191.3:38491)
0806 02:40:52.039569 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.039583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.039606 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:40:52.039683 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.039690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.039910 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.040131 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:52.040340 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.041428 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.042963 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.043791 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.044557 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.044720 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.044777 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.044874 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260806 02:40:52.045223 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.040013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.28.191.1:42503)
0806 02:40:52.040649 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.040662 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.040678 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:40:52.040734 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.040737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.040952 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.041476 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:52.041626 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.042718 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.043972 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.044600 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.044720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.044856 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.044944 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.045049 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":89,"threads_started":1}
I20260806 02:40:52.045399 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.039317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42503 (local address 127.0.0.1:53770)
0806 02:40:52.039854 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:52.039867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:52.039959 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:52.041133 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:52.041138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:52.041162 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:52.041284 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.041293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.042853 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.042859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.043845 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.043855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.044723 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.044727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.044812 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.044832 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.044905 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.044971 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":104,"threads_started":1}
I20260806 02:40:52.045704 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.039871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.28.191.2:42979)
0806 02:40:52.040297 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.040311 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.040460 (+ 149us) server_negotiation.cc:407] Connection header received
0806 02:40:52.040559 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.040564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.040739 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.040878 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:52.041300 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.042497 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.043939 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.044748 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.044846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.044964 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.045469 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.045561 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":94,"threads_started":1}
I20260806 02:40:52.045898 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.039631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42979 (local address 127.0.0.1:37864)
0806 02:40:52.040346 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:52.040359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:52.040461 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:52.041006 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:52.041008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:52.041024 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:52.041137 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.041143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.042622 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.042625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.043778 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.043790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.045032 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.045037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.045166 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.045192 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.045269 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.045331 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":94,"threads_started":1}
I20260806 02:40:52.048159 7071 tablet_service.cc:1467] Processing CreateTablet for tablet 5dba0c20d84c425aa3d747d032abd10d (DEFAULT_TABLE table=test-table [id=5766cb7e6ec34b1fa70937e046f068ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:52.048214 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 5dba0c20d84c425aa3d747d032abd10d (DEFAULT_TABLE table=test-table [id=5766cb7e6ec34b1fa70937e046f068ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:52.048216 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 5dba0c20d84c425aa3d747d032abd10d (DEFAULT_TABLE table=test-table [id=5766cb7e6ec34b1fa70937e046f068ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:52.049263 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba0c20d84c425aa3d747d032abd10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:52.049263 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba0c20d84c425aa3d747d032abd10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:52.049280 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dba0c20d84c425aa3d747d032abd10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:52.056473 7612 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Bootstrap starting.
I20260806 02:40:52.057662 7614 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Bootstrap starting.
I20260806 02:40:52.058720 7612 tablet_bootstrap.cc:654] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:52.059496 7612 log.cc:824] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:52.059530 7614 tablet_bootstrap.cc:654] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:52.060192 7614 log.cc:824] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:52.060874 7615 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Bootstrap starting.
I20260806 02:40:52.061481 7612 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: No bootstrap required, opened a new log
I20260806 02:40:52.061695 7612 ts_tablet_manager.cc:1397] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:40:52.062629 7615 tablet_bootstrap.cc:654] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:52.063356 7615 log.cc:824] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:52.064754 7614 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: No bootstrap required, opened a new log
I20260806 02:40:52.064893 7614 ts_tablet_manager.cc:1397] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260806 02:40:52.065346 7612 raft_consensus.cc:348] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.065596 7612 raft_consensus.cc:374] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:52.065659 7612 raft_consensus.cc:729] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7975b79b3b5d4bf7b27b0c11fedef25e, State: Initialized, Role: FOLLOWER
I20260806 02:40:52.066125 7612 consensus_queue.cc:260] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.066310 7615 tablet_bootstrap.cc:492] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: No bootstrap required, opened a new log
I20260806 02:40:52.066478 7615 ts_tablet_manager.cc:1397] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:40:52.067106 7612 ts_tablet_manager.cc:1428] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:40:52.067375 7167 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
I20260806 02:40:52.070811 7615 raft_consensus.cc:348] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.071055 7615 raft_consensus.cc:374] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:52.071116 7615 raft_consensus.cc:729] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25afa3a9cdf343b3977affe8f30b3ee8, State: Initialized, Role: FOLLOWER
I20260806 02:40:52.071520 7615 consensus_queue.cc:260] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.072247 7565 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
I20260806 02:40:52.072607 7615 ts_tablet_manager.cc:1428] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:40:52.073050 7614 raft_consensus.cc:348] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.073347 7614 raft_consensus.cc:374] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:52.074162 7614 raft_consensus.cc:729] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2a8490f83a4a3889ae7e5193e23922, State: Initialized, Role: FOLLOWER
I20260806 02:40:52.074586 7614 consensus_queue.cc:260] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.075364 7353 heartbeater.cc:503] Master 127.28.191.62:35097 was elected leader, sending a full tablet report...
I20260806 02:40:52.075512 7614 ts_tablet_manager.cc:1428] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
W20260806 02:40:52.075599 7169 tablet.cc:2369] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:52.118824 7623 raft_consensus.cc:482] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:52.119050 7623 raft_consensus.cc:504] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.120217 7623 leader_election.cc:288] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503), bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:52.124984 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.120396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42503 (local address 127.28.191.3:36379)
0806 02:40:52.120558 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:52.120576 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:52.120705 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:52.121016 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:52.121019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:52.121037 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:52.121198 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.121205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.122304 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.122307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.123592 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.123602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.124491 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.124494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.124608 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.124636 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.124716 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.124803 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260806 02:40:52.124995 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.120504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:36379 (local address 127.28.191.1:42503)
0806 02:40:52.120677 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.120683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.120700 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:40:52.120835 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.120841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.120898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.120997 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:52.121326 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.122170 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.123730 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.124374 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.124514 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.124588 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.124756 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.124855 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:40:52.124995 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.120929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:53817 (local address 127.28.191.2:42979)
0806 02:40:52.121112 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.121116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.121130 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:52.121183 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.121186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.121234 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.121308 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:52.121693 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.122493 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.123883 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.124478 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.124590 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.124645 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.124817 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.124881 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:40:52.124986 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.120699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42979 (local address 127.28.191.3:53817)
0806 02:40:52.120806 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:52.120823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:52.120905 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:52.121411 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:52.121413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:52.121428 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:52.121561 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.121567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.122631 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.122634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.123774 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:52.123781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.124594 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.124597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:52.124696 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.124723 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:40:52.124786 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:40:52.124833 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260806 02:40:52.125687 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba0c20d84c425aa3d747d032abd10d" candidate_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" is_pre_election: true
I20260806 02:40:52.125829 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba0c20d84c425aa3d747d032abd10d" candidate_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a8490f83a4a3889ae7e5193e23922" is_pre_election: true
I20260806 02:40:52.126036 7094 raft_consensus.cc:2454] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25afa3a9cdf343b3977affe8f30b3ee8 in term 0.
I20260806 02:40:52.126195 7290 raft_consensus.cc:2454] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25afa3a9cdf343b3977affe8f30b3ee8 in term 0.
I20260806 02:40:52.126579 7399 leader_election.cc:302] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25afa3a9cdf343b3977affe8f30b3ee8, 7975b79b3b5d4bf7b27b0c11fedef25e; no voters:
I20260806 02:40:52.126892 7623 raft_consensus.cc:2781] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:52.127031 7623 raft_consensus.cc:482] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:52.127104 7623 raft_consensus.cc:3037] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:52.128409 7623 raft_consensus.cc:504] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.128768 7623 leader_election.cc:288] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [CANDIDATE]: Term 1 election: Requested vote from peers 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503), bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979)
I20260806 02:40:52.129317 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba0c20d84c425aa3d747d032abd10d" candidate_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a8490f83a4a3889ae7e5193e23922"
I20260806 02:40:52.129420 7290 raft_consensus.cc:3037] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:52.130445 7290 raft_consensus.cc:2454] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25afa3a9cdf343b3977affe8f30b3ee8 in term 1.
I20260806 02:40:52.130697 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dba0c20d84c425aa3d747d032abd10d" candidate_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e"
I20260806 02:40:52.130724 7399 leader_election.cc:302] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25afa3a9cdf343b3977affe8f30b3ee8, bc2a8490f83a4a3889ae7e5193e23922; no voters:
I20260806 02:40:52.130817 7094 raft_consensus.cc:3037] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:52.130892 7623 raft_consensus.cc:2781] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:52.131124 7623 raft_consensus.cc:686] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 25afa3a9cdf343b3977affe8f30b3ee8, State: Running, Role: LEADER
I20260806 02:40:52.131721 7623 consensus_queue.cc:237] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } }
I20260806 02:40:52.132022 7094 raft_consensus.cc:2454] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25afa3a9cdf343b3977affe8f30b3ee8 in term 1.
I20260806 02:40:52.135210 5850 catalog_manager.cc:5495] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:52.167980 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.164712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.28.191.3:38491)
0806 02:40:52.164855 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.164858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.164871 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:52.165004 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.165008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.165112 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.165201 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:52.165404 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.165923 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.166761 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.166965 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.167049 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.167306 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:52.167683 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:52.167770 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.167836 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.167882 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:40:52.173760 7290 raft_consensus.cc:1261] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Refusing update from remote peer 25afa3a9cdf343b3977affe8f30b3ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:52.173760 7094 raft_consensus.cc:1261] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Refusing update from remote peer 25afa3a9cdf343b3977affe8f30b3ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:52.174474 7623 consensus_queue.cc:1035] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:52.174718 7625 consensus_queue.cc:1035] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:52.182744 7629 mvcc.cc:205] Tried to move back new op lower bound from 7315390677695913984 to 7315390677534613504. Current Snapshot: MvccSnapshot[applied={T|T < 7315390677695913984}]
I20260806 02:40:52.188112 7630 mvcc.cc:205] Tried to move back new op lower bound from 7315390677695913984 to 7315390677534613504. Current Snapshot: MvccSnapshot[applied={T|T < 7315390677695913984 or (T in {7315390677695913984})}]
I20260806 02:40:52.192088 5850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:52.192266 5850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:52.197566 5850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b10b88d1229a41088fd2232e3a46a931: Sending DeleteTablet for 3 replicas of tablet 5dba0c20d84c425aa3d747d032abd10d
I20260806 02:40:52.198292 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dba0c20d84c425aa3d747d032abd10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:52 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:43486
I20260806 02:40:52.198578 7071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dba0c20d84c425aa3d747d032abd10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:52 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:53770
I20260806 02:40:52.199863 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 5808
I20260806 02:40:52.199875 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dba0c20d84c425aa3d747d032abd10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:52 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:37864
I20260806 02:40:52.209408 7644 tablet_replica.cc:330] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: stopping tablet replica
I20260806 02:40:52.209780 7644 raft_consensus.cc:2229] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:52.210141 7644 raft_consensus.cc:2258] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:52.211894 7644 ts_tablet_manager.cc:1905] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:52.214645 7644 ts_tablet_manager.cc:1918] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:52.214751 7644 log.cc:1196] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/5dba0c20d84c425aa3d747d032abd10d
I20260806 02:40:52.215111 7644 ts_tablet_manager.cc:1939] T 5dba0c20d84c425aa3d747d032abd10d P bc2a8490f83a4a3889ae7e5193e23922: Deleting consensus metadata
W20260806 02:40:52.216375 7199 connection.cc:535] server connection from 127.0.0.1:37864 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:40:52.217028 7641 tablet_replica.cc:330] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: stopping tablet replica
I20260806 02:40:52.217315 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.205189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.28.191.61:44167)
0806 02:40:52.205329 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.205334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.205350 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:40:52.205395 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.205401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.205475 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.205622 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.207215 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.208042 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.209507 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.209960 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.210051 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.212416 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.212423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.212428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.212461 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.214388 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.215105 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.215111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.215116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.215221 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.215628 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.215634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.215639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.215893 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.216013 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.216923 (+ 910us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.217079 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:40:52.217773 7641 raft_consensus.cc:2229] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:52.218182 7641 raft_consensus.cc:2258] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:52.218628 7642 tablet_replica.cc:330] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: stopping tablet replica
I20260806 02:40:52.218900 7642 raft_consensus.cc:2229] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:52.219242 7642 raft_consensus.cc:2258] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:52.219810 7641 ts_tablet_manager.cc:1905] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:52.220567 7642 ts_tablet_manager.cc:1905] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:52.222636 7641 ts_tablet_manager.cc:1918] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:52.222749 7641 log.cc:1196] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/5dba0c20d84c425aa3d747d032abd10d
I20260806 02:40:52.223094 7641 ts_tablet_manager.cc:1939] T 5dba0c20d84c425aa3d747d032abd10d P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting consensus metadata
I20260806 02:40:52.223687 7642 ts_tablet_manager.cc:1918] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:52.223789 7642 log.cc:1196] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/5dba0c20d84c425aa3d747d032abd10d
I20260806 02:40:52.224129 7642 ts_tablet_manager.cc:1939] T 5dba0c20d84c425aa3d747d032abd10d P 7975b79b3b5d4bf7b27b0c11fedef25e: Deleting consensus metadata
W20260806 02:40:52.224231 7398 connection.cc:441] server connection from 127.0.0.1:43486 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43486 (request call id 1) could send its response
W20260806 02:40:52.225394 7008 connection.cc:441] server connection from 127.0.0.1:53770 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53770 (request call id 1) could send its response
I20260806 02:40:52.228688 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.205328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.28.191.60:35707)
0806 02:40:52.205862 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.205870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.219270 (+ 13400us) server_negotiation.cc:407] Connection header received
0806 02:40:52.219518 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.219521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.219580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.219640 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.220442 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.221307 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.222227 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.222579 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.222691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.224891 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.224895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.224897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.224925 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.226830 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.227323 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.227327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.227329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.227421 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.227742 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.227745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.227746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.227911 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.228052 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.228296 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.228421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421}
I20260806 02:40:52.232435 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.220822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.28.191.61:44167)
0806 02:40:52.220917 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.220921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.222988 (+ 2067us) server_negotiation.cc:407] Connection header received
0806 02:40:52.223310 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.223313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.223356 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.223409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.224870 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.225851 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.227125 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.227423 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.227530 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.229405 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.229409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.229412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.229436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.230866 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.231242 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.231245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.231247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.231347 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.231598 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.231601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.231603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.231812 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.231933 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.232182 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.232293 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:40:52.240759 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:52.221488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.28.191.60:35707)
0806 02:40:52.221628 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:52.221632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:52.232530 (+ 10898us) server_negotiation.cc:407] Connection header received
0806 02:40:52.232696 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:52.232700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:52.232744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:52.232797 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:52.233862 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.234630 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.235350 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:52.235699 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:52.235793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:52.237430 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:52.237436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:52.237439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:52.237470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:52.239174 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.239590 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.239594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.239595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.239676 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:52.239962 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:52.239966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:52.239968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:52.240173 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:52.240301 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:52.240487 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:40:52.240593 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:40:53.185429 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.184852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:49399)
0806 02:40:53.185230 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.185327 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
W20260806 02:40:53.185863 7565 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:35097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.186633 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.186090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:51905)
0806 02:40:53.186449 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.186537 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":79,"threads_started":1}
W20260806 02:40:53.186926 7353 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:35097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.190981 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.190619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:41575)
0806 02:40:53.190825 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.190896 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
W20260806 02:40:53.191308 7167 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:35097 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.697391 7658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b10b88d1229a41088fd2232e3a46a931)
I20260806 02:40:53.697671 7658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:53.697983 7659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b10b88d1229a41088fd2232e3a46a931)
I20260806 02:40:53.698148 7659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:53.698830 7658 leader_election.cc:288] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 07edf8dc8344473fa6768457546b2006 (127.28.191.61:44167)
I20260806 02:40:53.699362 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.698918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53026)
0806 02:40:53.699176 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.699260 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260806 02:40:53.699623 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.699088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53036)
0806 02:40:53.699433 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.699516 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":67,"threads_started":1}
W20260806 02:40:53.699651 6897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
W20260806 02:40:53.699774 6649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.699815 7659 leader_election.cc:288] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707)
I20260806 02:40:53.700532 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.700210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53052)
0806 02:40:53.700391 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.700447 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260806 02:40:53.700662 6897 leader_election.cc:334] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.701108 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.700532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53062)
0806 02:40:53.700936 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.700987 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:40:53.701265 6649 leader_election.cc:334] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.703506 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.699080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:44167 (local address 127.0.0.1:37058)
0806 02:40:53.699345 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.699360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:53.699465 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:53.700269 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:53.700273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:53.700291 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:53.700431 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:53.700439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.701611 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.701615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:53.702428 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:53.702437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.703101 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.703105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:53.703197 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:53.703221 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:40:53.703309 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:40:53.703381 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260806 02:40:53.703501 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.699197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.28.191.61:44167)
0806 02:40:53.700013 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:53.700017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:53.700031 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:53.700077 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:53.700081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:53.700127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:53.700539 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:53.700654 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.701486 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.702555 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.702968 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.703097 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:53.703156 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:53.703324 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:40:53.703392 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":66,"threads_started":1}
I20260806 02:40:53.703884 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.699668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:35707 (local address 127.0.0.1:49450)
0806 02:40:53.700169 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.700184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:53.700297 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:53.700877 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:53.700879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:53.700896 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:53.700967 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:53.700972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.701858 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.701863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:53.702802 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:53.702813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.703495 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.703499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:53.703593 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:53.703621 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:40:53.703692 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:40:53.703755 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":51,"threads_started":1}
I20260806 02:40:53.704032 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.699646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.28.191.60:35707)
0806 02:40:53.699856 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:53.699861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:53.700243 (+ 382us) server_negotiation.cc:407] Connection header received
0806 02:40:53.700647 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:53.700650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:53.700695 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:53.700754 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:53.701109 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.701736 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.702929 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:53.703363 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:53.703482 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:53.703550 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:53.703822 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 02:40:53.703899 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260806 02:40:53.704011 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59013bcdfcf2454a917f35d7e5c27900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07edf8dc8344473fa6768457546b2006" is_pre_election: true
I20260806 02:40:53.704206 6684 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59013bcdfcf2454a917f35d7e5c27900 in term 1.
I20260806 02:40:53.704442 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07edf8dc8344473fa6768457546b2006" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59013bcdfcf2454a917f35d7e5c27900" is_pre_election: true
I20260806 02:40:53.704566 6901 leader_election.cc:302] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07edf8dc8344473fa6768457546b2006, 59013bcdfcf2454a917f35d7e5c27900; no voters: b10b88d1229a41088fd2232e3a46a931
I20260806 02:40:53.704628 6932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07edf8dc8344473fa6768457546b2006 in term 1.
I20260806 02:40:53.704772 7658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:40:53.704833 7658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b10b88d1229a41088fd2232e3a46a931)
I20260806 02:40:53.704883 7658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:40:53.705116 6653 leader_election.cc:302] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07edf8dc8344473fa6768457546b2006, 59013bcdfcf2454a917f35d7e5c27900; no voters: b10b88d1229a41088fd2232e3a46a931
I20260806 02:40:53.705344 7659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:40:53.705417 7659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b10b88d1229a41088fd2232e3a46a931)
I20260806 02:40:53.705469 7659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:40:53.706117 7658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:53.706496 7658 leader_election.cc:288] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 election: Requested vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 07edf8dc8344473fa6768457546b2006 (127.28.191.61:44167)
I20260806 02:40:53.706661 7659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:53.706856 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.706647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53064)
0806 02:40:53.706740 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.706795 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 02:40:53.706840 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59013bcdfcf2454a917f35d7e5c27900" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07edf8dc8344473fa6768457546b2006"
I20260806 02:40:53.707003 6684 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59013bcdfcf2454a917f35d7e5c27900 in current term 2: Already voted for candidate 07edf8dc8344473fa6768457546b2006 in this term.
I20260806 02:40:53.707080 7659 leader_election.cc:288] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 election: Requested vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707)
I20260806 02:40:53.707427 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.707183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53076)
0806 02:40:53.707305 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.707360 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 02:40:53.707504 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07edf8dc8344473fa6768457546b2006" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59013bcdfcf2454a917f35d7e5c27900"
I20260806 02:40:53.707667 6932 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 07edf8dc8344473fa6768457546b2006 in current term 2: Already voted for candidate 59013bcdfcf2454a917f35d7e5c27900 in this term.
I20260806 02:40:53.707996 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.707496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53086)
0806 02:40:53.707908 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.707951 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260806 02:40:53.708137 6897 leader_election.cc:334] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.708158 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:53.707932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53098)
0806 02:40:53.708042 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:53.708096 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 02:40:53.708211 6897 leader_election.cc:302] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59013bcdfcf2454a917f35d7e5c27900; no voters: 07edf8dc8344473fa6768457546b2006, b10b88d1229a41088fd2232e3a46a931
W20260806 02:40:53.708254 6649 leader_election.cc:334] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:53.708331 6649 leader_election.cc:302] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07edf8dc8344473fa6768457546b2006; no voters: 59013bcdfcf2454a917f35d7e5c27900, b10b88d1229a41088fd2232e3a46a931
I20260806 02:40:53.708365 7658 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 02:40:53.708495 7659 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 02:40:54.187036 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:54.186545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:53635)
0806 02:40:54.186869 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:54.186935 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260806 02:40:54.188524 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:54.187626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:36677)
0806 02:40:54.188353 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:54.188426 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20260806 02:40:54.192193 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:54.191785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:47211)
0806 02:40:54.192058 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:54.192116 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260806 02:40:55.188413 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.187984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:50681)
0806 02:40:55.188250 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.188315 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260806 02:40:55.189711 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.189201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:57099)
0806 02:40:55.189555 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.189619 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":72,"threads_started":1}
I20260806 02:40:55.193218 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.192785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:50483)
0806 02:40:55.193020 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.193138 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260806 02:40:55.269541 7684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:55.269743 7684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:55.270277 7684 leader_election.cc:288] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707)
I20260806 02:40:55.270903 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.270462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53334)
0806 02:40:55.270727 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.270810 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260806 02:40:55.270896 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07edf8dc8344473fa6768457546b2006" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59013bcdfcf2454a917f35d7e5c27900" is_pre_election: true
I20260806 02:40:55.271068 6932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07edf8dc8344473fa6768457546b2006 in term 2.
I20260806 02:40:55.271422 6653 leader_election.cc:302] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07edf8dc8344473fa6768457546b2006, 59013bcdfcf2454a917f35d7e5c27900; no voters:
I20260806 02:40:55.271634 7684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 02:40:55.271687 7684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:55.271740 7684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:40:55.271826 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.271570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53344)
0806 02:40:55.271696 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.271756 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:40:55.271955 6649 leader_election.cc:334] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:55.272869 7684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:55.273262 7684 leader_election.cc:288] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 election: Requested vote from peers b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097), 59013bcdfcf2454a917f35d7e5c27900 (127.28.191.60:35707)
I20260806 02:40:55.273600 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.273381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53346)
0806 02:40:55.273476 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.273537 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 02:40:55.273631 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07edf8dc8344473fa6768457546b2006" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59013bcdfcf2454a917f35d7e5c27900"
I20260806 02:40:55.273772 6932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:40:55.274276 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.274051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53360)
0806 02:40:55.274165 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.274214 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:40:55.274394 6649 leader_election.cc:334] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
I20260806 02:40:55.274984 6932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07edf8dc8344473fa6768457546b2006 in term 3.
I20260806 02:40:55.275290 6653 leader_election.cc:302] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07edf8dc8344473fa6768457546b2006, 59013bcdfcf2454a917f35d7e5c27900; no voters: b10b88d1229a41088fd2232e3a46a931
I20260806 02:40:55.275466 7684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:40:55.275630 7684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [term 3 LEADER]: Becoming Leader. State: Replica: 07edf8dc8344473fa6768457546b2006, State: Running, Role: LEADER
I20260806 02:40:55.275897 7684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } }
I20260806 02:40:55.276942 7688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07edf8dc8344473fa6768457546b2006. Latest consensus state: current_term: 3 leader_uuid: "07edf8dc8344473fa6768457546b2006" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:55.277143 7688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:55.277451 7691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:55.279944 7691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:55.280367 7691 catalog_manager.cc:1259] Loaded cluster ID: 1d2d2c184bb645ec80bf9ebfef440b71
I20260806 02:40:55.280413 7691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:55.280987 7691 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:55.281410 7691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Loaded TSK: 0
I20260806 02:40:55.282116 7691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:40:55.298902 6669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:55.302673 6669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:55.305879 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.305594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53378)
0806 02:40:55.305726 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.305778 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:40:55.306072 6932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [term 3 FOLLOWER]: Refusing update from remote peer 07edf8dc8344473fa6768457546b2006: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 02:40:55.306627 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.306392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.0.0.1:53388)
0806 02:40:55.306498 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.306555 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 02:40:55.306622 7688 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 02:40:55.306847 6649 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 -> Peer b10b88d1229a41088fd2232e3a46a931 (127.28.191.62:35097): Couldn't send request to peer b10b88d1229a41088fd2232e3a46a931. Status: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:40:55.308141 7693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07edf8dc8344473fa6768457546b2006. Latest consensus state: current_term: 3 leader_uuid: "07edf8dc8344473fa6768457546b2006" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:55.308262 7693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:55.309417 7688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "07edf8dc8344473fa6768457546b2006" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:55.309536 7688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:55.309917 7693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "07edf8dc8344473fa6768457546b2006" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b10b88d1229a41088fd2232e3a46a931" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35097 } } peers { permanent_uuid: "07edf8dc8344473fa6768457546b2006" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 44167 } attrs { promote: false } } peers { permanent_uuid: "59013bcdfcf2454a917f35d7e5c27900" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 35707 } attrs { promote: false } } }
I20260806 02:40:55.310039 7693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59013bcdfcf2454a917f35d7e5c27900 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:40:55.319722 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.314996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42503 (local address 127.0.0.1:53784)
0806 02:40:55.315121 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.315139 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:55.315238 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:55.315958 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:55.315961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:55.315977 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:55.316112 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.316119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.317221 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.317224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.318421 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.318432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.319349 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.319352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.319443 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.319472 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:40:55.319518 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 02:40:55.319586 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260806 02:40:55.320371 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.315409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.28.191.1:42503)
0806 02:40:55.315683 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.315690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.315709 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:40:55.315770 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.315773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.315827 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.316231 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:55.316360 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.317091 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.318561 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.319224 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.320028 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.320094 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.320128 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.320210 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260806 02:40:55.320612 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.316162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.28.191.3:38491)
0806 02:40:55.316433 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.316439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.317040 (+ 601us) server_negotiation.cc:407] Connection header received
0806 02:40:55.317192 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.317196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.317250 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.317354 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:55.317637 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.318127 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.319398 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.319857 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.319962 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.320023 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.320404 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.320482 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
I20260806 02:40:55.320612 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.315629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:38491 (local address 127.0.0.1:43498)
0806 02:40:55.316949 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.316962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:55.317094 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:55.317363 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:55.317366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:55.317383 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:55.317512 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.317522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.318233 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.318239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.319298 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.319309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.320085 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.320088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.320239 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.320293 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:40:55.320385 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:40:55.320460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"mutex_wait_us":122,"thread_start_us":92,"threads_started":1}
I20260806 02:40:55.320794 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.315273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42979 (local address 127.0.0.1:37876)
0806 02:40:55.315504 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.315764 (+ 260us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:55.315862 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:55.316480 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:55.316483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:55.316499 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:55.316586 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.316598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.317842 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.317845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.319058 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.319068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.320074 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.320078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.320238 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.320290 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 02:40:55.320360 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:40:55.320443 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260806 02:40:55.321137 7071 tablet_service.cc:1467] Processing CreateTablet for tablet 2771a6d0510a426c9d58018f68b81466 (DEFAULT_TABLE table=test-table [id=47cf1a0f35914aab8f26c6d794756157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:55.321219 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.315657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.28.191.2:42979)
0806 02:40:55.316162 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.316165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.316176 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:40:55.316221 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.316223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.316261 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.316361 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:55.316832 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.317710 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.319199 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.319823 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.319938 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.320017 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.320538 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.320616 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":107,"threads_started":1}
I20260806 02:40:55.321403 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 2771a6d0510a426c9d58018f68b81466 (DEFAULT_TABLE table=test-table [id=47cf1a0f35914aab8f26c6d794756157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:55.321486 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2771a6d0510a426c9d58018f68b81466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:55.321688 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2771a6d0510a426c9d58018f68b81466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:55.321769 7260 tablet_service.cc:1467] Processing CreateTablet for tablet 2771a6d0510a426c9d58018f68b81466 (DEFAULT_TABLE table=test-table [id=47cf1a0f35914aab8f26c6d794756157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:55.322013 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2771a6d0510a426c9d58018f68b81466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:55.323788 7711 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: Bootstrap starting.
I20260806 02:40:55.324314 7712 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Bootstrap starting.
I20260806 02:40:55.324716 7711 tablet_bootstrap.cc:654] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:55.324903 7714 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Bootstrap starting.
I20260806 02:40:55.325297 7712 tablet_bootstrap.cc:654] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:55.325865 7714 tablet_bootstrap.cc:654] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:55.326851 7712 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: No bootstrap required, opened a new log
I20260806 02:40:55.326951 7712 ts_tablet_manager.cc:1397] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:55.327198 7714 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: No bootstrap required, opened a new log
I20260806 02:40:55.327219 7711 tablet_bootstrap.cc:492] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: No bootstrap required, opened a new log
I20260806 02:40:55.327286 7714 ts_tablet_manager.cc:1397] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:55.327347 7711 ts_tablet_manager.cc:1397] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:40:55.327409 7712 raft_consensus.cc:348] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.327533 7712 raft_consensus.cc:374] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:55.327570 7712 raft_consensus.cc:729] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25afa3a9cdf343b3977affe8f30b3ee8, State: Initialized, Role: FOLLOWER
I20260806 02:40:55.327709 7714 raft_consensus.cc:348] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.327822 7714 raft_consensus.cc:374] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:55.327790 7712 consensus_queue.cc:260] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.327857 7714 raft_consensus.cc:729] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2a8490f83a4a3889ae7e5193e23922, State: Initialized, Role: FOLLOWER
I20260806 02:40:55.327875 7711 raft_consensus.cc:348] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.327997 7711 raft_consensus.cc:374] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:55.328042 7711 raft_consensus.cc:729] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7975b79b3b5d4bf7b27b0c11fedef25e, State: Initialized, Role: FOLLOWER
I20260806 02:40:55.327981 7714 consensus_queue.cc:260] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.328174 7711 consensus_queue.cc:260] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.328259 7712 ts_tablet_manager.cc:1428] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:55.328333 7714 ts_tablet_manager.cc:1428] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:55.328562 7711 ts_tablet_manager.cc:1428] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:40:55.329227 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.328888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:60801)
0806 02:40:55.329093 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.329153 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:40:55.329805 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.329507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:58015)
0806 02:40:55.329685 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.329736 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:40:55.329873 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.329647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:57263)
0806 02:40:55.329775 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.329818 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 02:40:55.331295 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.331020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:56617)
0806 02:40:55.331181 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.331232 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 02:40:55.331943 7168 heartbeater.cc:503] Master 127.28.191.61:44167 was elected leader, sending a full tablet report...
I20260806 02:40:55.333194 7354 heartbeater.cc:503] Master 127.28.191.61:44167 was elected leader, sending a full tablet report...
I20260806 02:40:55.333297 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.333021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:43421)
0806 02:40:55.333167 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.333225 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:40:55.361733 7726 raft_consensus.cc:482] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:55.361943 7726 raft_consensus.cc:504] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.363631 7726 leader_election.cc:288] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979), 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:55.368547 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.363798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:33301 (local address 127.28.191.3:38491)
0806 02:40:55.363936 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.363941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.364587 (+ 646us) server_negotiation.cc:407] Connection header received
0806 02:40:55.364647 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.364650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.364696 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.364769 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:55.365049 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.365706 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367180 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.367783 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367907 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.367960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.368334 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.368411 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:40:55.368542 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.363798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:47759 (local address 127.28.191.2:42979)
0806 02:40:55.363928 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.363934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.363952 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:40:55.364114 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.364119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.364171 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.364255 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:55.364570 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.365388 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367180 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.367783 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367907 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.367960 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.368334 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.368411 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:40:55.368564 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.363596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42979 (local address 127.28.191.1:47759)
0806 02:40:55.363816 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.363834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:55.363975 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:55.364284 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:55.364287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:55.364308 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:55.364438 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.364444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.365531 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.365538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.367033 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.367047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.367968 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.368184 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.368211 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:40:55.368300 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:40:55.368397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":132}
I20260806 02:40:55.368573 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.363596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:38491 (local address 127.28.191.1:33301)
0806 02:40:55.364257 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.364272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:55.364382 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:55.364803 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:55.364806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:55.364822 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:55.364923 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.364928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.365810 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.365814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.367047 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:55.367054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.367968 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.367972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:55.368164 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.368210 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:40:55.368300 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:40:55.368397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":315,"thread_start_us":147,"threads_started":1}
I20260806 02:40:55.369272 7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2771a6d0510a426c9d58018f68b81466" candidate_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" is_pre_election: true
I20260806 02:40:55.369314 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2771a6d0510a426c9d58018f68b81466" candidate_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a8490f83a4a3889ae7e5193e23922" is_pre_election: true
I20260806 02:40:55.369460 7290 raft_consensus.cc:2454] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7975b79b3b5d4bf7b27b0c11fedef25e in term 0.
I20260806 02:40:55.369526 7488 raft_consensus.cc:2454] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7975b79b3b5d4bf7b27b0c11fedef25e in term 0.
I20260806 02:40:55.369863 7008 leader_election.cc:302] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7975b79b3b5d4bf7b27b0c11fedef25e, bc2a8490f83a4a3889ae7e5193e23922; no voters:
I20260806 02:40:55.370174 7726 raft_consensus.cc:2781] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:55.370239 7726 raft_consensus.cc:482] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:55.370277 7726 raft_consensus.cc:3037] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:55.371321 7726 raft_consensus.cc:504] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.371676 7726 leader_election.cc:288] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [CANDIDATE]: Term 1 election: Requested vote from peers bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979), 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491)
I20260806 02:40:55.372010 7290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2771a6d0510a426c9d58018f68b81466" candidate_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2a8490f83a4a3889ae7e5193e23922"
I20260806 02:40:55.372138 7290 raft_consensus.cc:3037] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:55.372157 7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2771a6d0510a426c9d58018f68b81466" candidate_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25afa3a9cdf343b3977affe8f30b3ee8"
I20260806 02:40:55.372315 7488 raft_consensus.cc:3037] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:55.373179 7290 raft_consensus.cc:2454] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7975b79b3b5d4bf7b27b0c11fedef25e in term 1.
I20260806 02:40:55.373459 7008 leader_election.cc:302] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7975b79b3b5d4bf7b27b0c11fedef25e, bc2a8490f83a4a3889ae7e5193e23922; no voters:
I20260806 02:40:55.373593 7488 raft_consensus.cc:2454] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7975b79b3b5d4bf7b27b0c11fedef25e in term 1.
I20260806 02:40:55.373651 7726 raft_consensus.cc:2781] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:55.373904 7726 raft_consensus.cc:686] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 LEADER]: Becoming Leader. State: Replica: 7975b79b3b5d4bf7b27b0c11fedef25e, State: Running, Role: LEADER
I20260806 02:40:55.374176 7726 consensus_queue.cc:237] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } }
I20260806 02:40:55.374637 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.374375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:45077)
0806 02:40:55.374507 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.374571 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 02:40:55.376698 6667 catalog_manager.cc:5495] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1). New cstate: current_term: 1 leader_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7975b79b3b5d4bf7b27b0c11fedef25e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:55.400902 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.397634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.28.191.1:42503)
0806 02:40:55.397752 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:55.397755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:55.397769 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:55.397904 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:55.397906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:55.397959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:55.398053 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:55.398274 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.398770 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.399552 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:55.399763 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:55.399841 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:55.400073 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:55.400519 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:55.400607 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:55.400752 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 02:40:55.400799 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:40:55.403882 7290 raft_consensus.cc:1261] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Refusing update from remote peer 7975b79b3b5d4bf7b27b0c11fedef25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:55.404121 7488 raft_consensus.cc:1261] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Refusing update from remote peer 7975b79b3b5d4bf7b27b0c11fedef25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:55.404546 7728 consensus_queue.cc:1035] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2a8490f83a4a3889ae7e5193e23922" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:55.404754 7726 consensus_queue.cc:1035] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "25afa3a9cdf343b3977affe8f30b3ee8" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:55.406179 7566 heartbeater.cc:503] Master 127.28.191.61:44167 was elected leader, sending a full tablet report...
I20260806 02:40:55.407759 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.407452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:47929)
0806 02:40:55.407607 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.407677 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 02:40:55.408105 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.407795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:53633)
0806 02:40:55.407963 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.408031 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:40:55.409783 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.409466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:43507)
0806 02:40:55.409636 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.409692 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 02:40:55.411649 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.411448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.1:59631)
0806 02:40:55.411544 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.411593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 02:40:55.412153 7734 mvcc.cc:205] Tried to move back new op lower bound from 7315390690928664576 to 7315390690815434752. Current Snapshot: MvccSnapshot[applied={T|T < 7315390690928664576 or (T in {7315390690928664576})}]
I20260806 02:40:55.413813 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.413528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.2:57457)
0806 02:40:55.413667 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.413733 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:40:55.416158 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.415936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35097 (local address 127.28.191.3:58697)
0806 02:40:55.416034 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.416092 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 02:40:55.419057 6667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:55.419205 6667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:55.423267 6667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07edf8dc8344473fa6768457546b2006: Sending DeleteTablet for 3 replicas of tablet 2771a6d0510a426c9d58018f68b81466
I20260806 02:40:55.423913 7071 tablet_service.cc:1514] Processing DeleteTablet for tablet 2771a6d0510a426c9d58018f68b81466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:55 UTC) from {username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:53784
I20260806 02:40:55.423908 7260 tablet_service.cc:1514] Processing DeleteTablet for tablet 2771a6d0510a426c9d58018f68b81466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:55 UTC) from {username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:37876
I20260806 02:40:55.424151 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 2771a6d0510a426c9d58018f68b81466 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:55 UTC) from {username='oryx', principal='oryx/127.28.191.61@KRBTEST.COM'} at 127.0.0.1:43498
I20260806 02:40:55.424264 7748 tablet_replica.cc:330] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: stopping tablet replica
I20260806 02:40:55.424407 7748 raft_consensus.cc:2229] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:55.424474 7755 tablet_replica.cc:330] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: stopping tablet replica
I20260806 02:40:55.424580 7755 raft_consensus.cc:2229] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:55.424692 7748 raft_consensus.cc:2258] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:55.424821 7755 raft_consensus.cc:2258] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:55.425346 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 6869
I20260806 02:40:55.425451 7748 ts_tablet_manager.cc:1905] T 2771a6d0510a426c9d58018f68b81466 P 7975b79b3b5d4bf7b27b0c11fedef25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:55.426054 7752 tablet_replica.cc:330] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: stopping tablet replica
I20260806 02:40:55.426201 7752 raft_consensus.cc:2229] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:55.426347 7752 raft_consensus.cc:2258] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:55.431135 7752 ts_tablet_manager.cc:1905] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:55.431779 7755 ts_tablet_manager.cc:1905] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:55.433892 7755 ts_tablet_manager.cc:1918] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:55.433991 7755 log.cc:1196] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/2771a6d0510a426c9d58018f68b81466
I20260806 02:40:55.434306 7755 ts_tablet_manager.cc:1939] T 2771a6d0510a426c9d58018f68b81466 P 25afa3a9cdf343b3977affe8f30b3ee8: Deleting consensus metadata
I20260806 02:40:55.434701 7752 ts_tablet_manager.cc:1918] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:55.434796 7752 log.cc:1196] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/2771a6d0510a426c9d58018f68b81466
I20260806 02:40:55.435039 6653 catalog_manager.cc:4841] TS 25afa3a9cdf343b3977affe8f30b3ee8 (127.28.191.3:38491): tablet 2771a6d0510a426c9d58018f68b81466 (table test-table [id=47cf1a0f35914aab8f26c6d794756157]) successfully deleted
I20260806 02:40:55.435317 7752 ts_tablet_manager.cc:1939] T 2771a6d0510a426c9d58018f68b81466 P bc2a8490f83a4a3889ae7e5193e23922: Deleting consensus metadata
I20260806 02:40:55.436198 6656 catalog_manager.cc:4841] TS bc2a8490f83a4a3889ae7e5193e23922 (127.28.191.2:42979): tablet 2771a6d0510a426c9d58018f68b81466 (table test-table [id=47cf1a0f35914aab8f26c6d794756157]) successfully deleted
I20260806 02:40:55.440732 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 7175
I20260806 02:40:55.440834 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:55.440535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42503 (local address 127.0.0.1:53798)
0806 02:40:55.440684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:55.440740 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:40:55.442689 6656 catalog_manager.cc:4584] TS 7975b79b3b5d4bf7b27b0c11fedef25e (127.28.191.1:42503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2771a6d0510a426c9d58018f68b81466: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42503: connect: Connection refused (error 111)
I20260806 02:40:55.448860 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 7362
I20260806 02:40:55.468947 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 6477
I20260806 02:40:55.480327 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 6621
2026-08-06T02:40:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7073 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:55 dist-test-slave-tfpf krb5kdc[7760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:55 dist-test-slave-tfpf krb5kdc[7760](info): set up 2 sockets
Aug 06 02:40:55 dist-test-slave-tfpf krb5kdc[7760](info): commencing operation
krb5kdc: starting...
W20260806 02:40:56.708029 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:40:56 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:40:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab.
Entry for principal oryx/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab.
I20260806 02:40:56.836374 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:41343
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab
--principal=oryx/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:56.950176 7785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:56.950424 7785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:56.950465 7785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:56.954489 7785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:40:56.954542 7785 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:40:56.954563 7785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:56.954584 7785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:56.954655 7785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:40:56.954682 7785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:40:56.960009 7785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:41343
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.62.keytab
--principal=oryx/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:56.960991 7785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:56.961999 7785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:56.970641 7791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:56.970652 7793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:56.971103 7790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:56.971159 7785 server_base.cc:1034] running on GCE node
I20260806 02:40:56.971709 7785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:56.972986 7785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:56.974222 7785 hybrid_clock.cc:648] HybridClock initialized: now 1785984056974138 us; error 100 us; skew 500 ppm
Aug 06 02:40:56 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:56.977834 7785 init.cc:377] Logged in from keytab as oryx/127.28.191.62@KRBTEST.COM (short username oryx)
I20260806 02:40:56.978713 7785 webserver.cc:460] Webserver started at http://127.28.191.62:42915/ using document root <none> and password file <none>
I20260806 02:40:56.979243 7785 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:56.979303 7785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:56.979465 7785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:56.981698 7785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "0bac4e6f3a0d49e99c8068308a9bbb68"
format_stamp: "Formatted at 2026-08-06 02:40:56 on dist-test-slave-tfpf"
server_key: "84e97f3b7da7205522de09196baa52aa"
server_key_iv: "4cc84eb4ed3b8a0437aca632406a7999"
server_key_version: "encryptionkey@0"
I20260806 02:40:56.982335 7785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "0bac4e6f3a0d49e99c8068308a9bbb68"
format_stamp: "Formatted at 2026-08-06 02:40:56 on dist-test-slave-tfpf"
server_key: "84e97f3b7da7205522de09196baa52aa"
server_key_iv: "4cc84eb4ed3b8a0437aca632406a7999"
server_key_version: "encryptionkey@0"
I20260806 02:40:56.986117 7785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:40:56.988502 7803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:56.989950 7785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:56.990128 7785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "0bac4e6f3a0d49e99c8068308a9bbb68"
format_stamp: "Formatted at 2026-08-06 02:40:56 on dist-test-slave-tfpf"
server_key: "84e97f3b7da7205522de09196baa52aa"
server_key_iv: "4cc84eb4ed3b8a0437aca632406a7999"
server_key_version: "encryptionkey@0"
I20260806 02:40:56.990267 7785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:57.005677 7785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:57.009150 7785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:57.009362 7785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:57.017500 7785 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:41343
I20260806 02:40:57.017515 7894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:41343 every 8 connection(s)
I20260806 02:40:57.018445 7785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:40:57.021194 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.023342 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 7785
I20260806 02:40:57.023492 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:40:57.023838 29436 external_mini_cluster.cc:1428] Setting key aec35511578d0a7f08f4233341807880
I20260806 02:40:57.026901 7895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Bootstrap starting.
I20260806 02:40:57.029352 7895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.030187 7895 log.cc:824] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:57.032476 7895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: No bootstrap required, opened a new log
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:57.037164 7895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } }
I20260806 02:40:57.037492 7895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.037570 7895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bac4e6f3a0d49e99c8068308a9bbb68, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.038041 7895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } }
I20260806 02:40:57.038206 7895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:40:57.038273 7895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:40:57.038375 7895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:57.038571 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.025242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.28.191.62:41343)
0806 02:40:57.025627 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.025642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.025677 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:40:57.026287 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.026304 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.026605 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.026868 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.027754 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.028717 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.029886 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.030410 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.030525 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.033431 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.033438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.033453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.033484 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.035407 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.035859 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.035865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.035871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.036016 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.036296 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.036299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.036301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.036828 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.037033 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.037325 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.037627 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260806 02:40:57.039307 7895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } }
I20260806 02:40:57.039638 7895 leader_election.cc:302] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bac4e6f3a0d49e99c8068308a9bbb68; no voters:
I20260806 02:40:57.039948 7895 leader_election.cc:288] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:40:57.040033 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:57.040228 7900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [term 1 LEADER]: Becoming Leader. State: Replica: 0bac4e6f3a0d49e99c8068308a9bbb68, State: Running, Role: LEADER
I20260806 02:40:57.040905 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } }
I20260806 02:40:57.041213 7895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:40:57.042482 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } } }
I20260806 02:40:57.042609 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:57.042896 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bac4e6f3a0d49e99c8068308a9bbb68. Latest consensus state: current_term: 1 leader_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bac4e6f3a0d49e99c8068308a9bbb68" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41343 } } }
I20260806 02:40:57.043066 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68 [sys.catalog]: This master's current role is: LEADER
I20260806 02:40:57.043094 7915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:40:57.046383 7915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:40:57.051587 7915 catalog_manager.cc:1347] Generated new cluster ID: 4919c2c4b39543b4a0e77b8a200d62ad
I20260806 02:40:57.051661 7915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:40:57.069124 7915 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:40:57.070103 7915 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:40:57.074910 7915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Generated new TSK 0
I20260806 02:40:57.075553 7915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
I20260806 02:40:57.105581 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41343
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:57.217682 7930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:57.217926 7930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:57.217999 7930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:57.221954 7930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:57.222002 7930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:57.222074 7930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:57.222144 7930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:40:57.227151 7930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:57.228091 7930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:57.229110 7930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:57.238472 7940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:57.238523 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:57.238482 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:57.238631 7930 server_base.cc:1034] running on GCE node
I20260806 02:40:57.239187 7930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:57.239786 7930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:57.240978 7930 hybrid_clock.cc:648] HybridClock initialized: now 1785984057240964 us; error 31 us; skew 500 ppm
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:57.244066 7930 init.cc:377] Logged in from keytab as oryx/127.28.191.1@KRBTEST.COM (short username oryx)
I20260806 02:40:57.244839 7930 webserver.cc:460] Webserver started at http://127.28.191.1:43837/ using document root <none> and password file <none>
I20260806 02:40:57.245426 7930 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:57.245491 7930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:57.245659 7930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:57.247942 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "64145fe1927c47528bf330fda7f444b2"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "96040391ba389fa8bec09acf50968687"
server_key_iv: "f47613ad67fff86abf4c6183594c6e71"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.248515 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "64145fe1927c47528bf330fda7f444b2"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "96040391ba389fa8bec09acf50968687"
server_key_iv: "f47613ad67fff86abf4c6183594c6e71"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.252342 7930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 02:40:57.254841 7949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.256035 7930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:57.256138 7930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "64145fe1927c47528bf330fda7f444b2"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "96040391ba389fa8bec09acf50968687"
server_key_iv: "f47613ad67fff86abf4c6183594c6e71"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.256225 7930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:57.270088 7930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:57.273527 7930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:57.273694 7930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:57.275179 7930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:57.275260 7930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.275377 7930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:57.275410 7930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:57.287853 7930 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:37411
I20260806 02:40:57.287854 8118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:37411 every 8 connection(s)
I20260806 02:40:57.288966 7930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:40:57.290280 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.275876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:37759 (local address 127.28.191.62:41343)
0806 02:40:57.276040 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.276047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.276888 (+ 841us) server_negotiation.cc:407] Connection header received
0806 02:40:57.277866 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.277870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.277935 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.278048 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.279487 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.280226 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.281729 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.282063 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.282188 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.286018 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.286023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.286026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.286061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.287914 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.288465 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.288478 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.288480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.288589 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.289012 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.289017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.289019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.289312 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.289447 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.290030 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.290157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:40:57.290995 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.276249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.1:37759)
0806 02:40:57.276696 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.276764 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.277635 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.278364 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.278371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.278877 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.279332 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.279345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.280381 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.280385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.281598 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.281612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.282227 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.282232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.282418 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.283066 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.283092 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.285841 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.288060 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.288067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.288080 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.288337 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.288730 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.288732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.288734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.288879 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.289451 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.289459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.289737 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.289928 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.290199 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260806 02:40:57.291796 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 7930
I20260806 02:40:57.291962 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:40:57.292243 29436 external_mini_cluster.cc:1428] Setting key bc2e29bb9012b58294eab0e57abcacad
I20260806 02:40:57.292377 8119 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41343
I20260806 02:40:57.292522 8119 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:57.293790 8119 heartbeater.cc:511] Master 127.28.191.62:41343 requested a full tablet report, sending...
I20260806 02:40:57.295445 7824 ts_manager.cc:194] Registered new tserver with Master: 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411)
I20260806 02:40:57.297050 7824 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:37759
WARNING: no policy specified for oryx/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:40:57.305198 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.294680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:44229 (local address 127.28.191.62:41343)
0806 02:40:57.294823 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.294830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.297419 (+ 2589us) server_negotiation.cc:407] Connection header received
0806 02:40:57.297584 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.297594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.297637 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.297731 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.298410 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.299175 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.300186 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.300429 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.300547 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.301770 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.301774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.301777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.301804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.303456 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.303914 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.303917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.303919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.304009 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.304343 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.304348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.304350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.304528 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.304638 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.304905 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.305019 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:40:57.305207 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.294572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.1:44229)
0806 02:40:57.297315 (+ 2743us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.297339 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.297470 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.297872 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.297874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.298118 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.298253 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.298262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.299363 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.299366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.300049 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.300059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.300561 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.300566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.300671 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.301265 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.301278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.301623 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.303572 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.303576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.303578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.303802 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.304149 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.304152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.304155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.304245 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.304632 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.304635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.304734 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.304846 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.304985 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
I20260806 02:40:57.320070 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41343
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:57.431981 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:57.432222 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:57.432264 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:57.436399 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:57.436455 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:57.436492 8132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:57.436560 8132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:40:57.441774 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:57.442780 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:57.443711 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:57.453145 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:57.453311 8132 server_base.cc:1034] running on GCE node
W20260806 02:40:57.453138 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:57.453145 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:57.453850 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:57.454387 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:57.455554 8132 hybrid_clock.cc:648] HybridClock initialized: now 1785984057455525 us; error 50 us; skew 500 ppm
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:57.458707 8132 init.cc:377] Logged in from keytab as oryx/127.28.191.2@KRBTEST.COM (short username oryx)
I20260806 02:40:57.459695 8132 webserver.cc:460] Webserver started at http://127.28.191.2:40333/ using document root <none> and password file <none>
I20260806 02:40:57.460219 8132 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:57.460285 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:57.460455 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:57.462534 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "8d35be173cd140b1a6df29da1c45155a"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "e4efdaacc38e8bde559d90801b77076c"
server_key_iv: "231a50c35a33cd29026a5fbf4408dcc2"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.463090 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "8d35be173cd140b1a6df29da1c45155a"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "e4efdaacc38e8bde559d90801b77076c"
server_key_iv: "231a50c35a33cd29026a5fbf4408dcc2"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.466866 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:40:57.469482 8155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.470676 8132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:40:57.470778 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "8d35be173cd140b1a6df29da1c45155a"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "e4efdaacc38e8bde559d90801b77076c"
server_key_iv: "231a50c35a33cd29026a5fbf4408dcc2"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.470916 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:57.502696 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:57.505935 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:57.506160 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:57.507718 8132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:57.507769 8132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.507810 8132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:57.507853 8132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:57.519240 8132 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:34327
I20260806 02:40:57.519310 8326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:34327 every 8 connection(s)
I20260806 02:40:57.520115 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:40:57.521821 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.508508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:35697 (local address 127.28.191.62:41343)
0806 02:40:57.508668 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.508673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.509658 (+ 985us) server_negotiation.cc:407] Connection header received
0806 02:40:57.510681 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.510686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.510750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.510810 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.512553 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.513316 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.514639 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.514973 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.515114 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.517899 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.517903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.517907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.517939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.519680 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.520218 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.520222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.520224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.520310 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.520740 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.520744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.520746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.520980 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.521128 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.521593 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.521699 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:40:57.522564 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.508960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.2:35697)
0806 02:40:57.509481 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.509543 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.510395 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.511197 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.511204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.511640 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.512295 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.512313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.513449 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.513452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.514496 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.514507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.515163 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.515167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.515346 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.515933 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.515950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.517726 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.519810 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.519816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.519830 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.520104 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.520414 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.520419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.520421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.520618 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.521146 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.521152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.521368 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.521531 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.521771 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":136,"threads_started":1}
I20260806 02:40:57.523810 8328 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41343
I20260806 02:40:57.524083 8328 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:57.524664 8328 heartbeater.cc:511] Master 127.28.191.62:41343 requested a full tablet report, sending...
I20260806 02:40:57.525866 7826 ts_manager.cc:194] Registered new tserver with Master: 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327)
I20260806 02:40:57.526660 7826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:35697
I20260806 02:40:57.526854 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 8132
I20260806 02:40:57.527011 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:40:57.527385 29436 external_mini_cluster.cc:1428] Setting key cec5f086e9a4a1f47fb7baaa315d2d46
I20260806 02:40:57.534503 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.526368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:48949 (local address 127.28.191.62:41343)
0806 02:40:57.526500 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.526505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.526725 (+ 220us) server_negotiation.cc:407] Connection header received
0806 02:40:57.526856 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.526860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.526914 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.527005 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.527653 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.528310 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.529487 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.529700 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.529779 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.531069 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.531074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.531078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.531111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.532772 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.533323 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.533326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.533327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.533400 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.533722 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.533725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.533726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.533918 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.533996 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.534306 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.534394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:40:57.534723 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.526304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.2:48949)
0806 02:40:57.526624 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.526639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.526751 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.527136 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.527140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.527356 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.527508 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.527517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.528440 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.528442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.529343 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.529358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.529816 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.529823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.529951 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.530417 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.530438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.530887 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.532915 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.532921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.532926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.533227 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.533494 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.533500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.533505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.533630 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.534022 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.534025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.534136 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.534256 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.534423 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for oryx/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
I20260806 02:40:57.559391 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41343
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:40:57.671181 8343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:40:57.671411 8343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:40:57.671453 8343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:40:57.675702 8343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:40:57.675757 8343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:40:57.675824 8343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:40:57.675917 8343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:40:57.680871 8343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.191.62:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:40:57.681864 8343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:40:57.682847 8343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:40:57.692713 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:57.692783 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:40:57.692772 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:40:57.692948 8343 server_base.cc:1034] running on GCE node
I20260806 02:40:57.693401 8343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:40:57.694114 8343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:40:57.695297 8343 hybrid_clock.cc:648] HybridClock initialized: now 1785984057695280 us; error 24 us; skew 500 ppm
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:40:57.698016 8343 init.cc:377] Logged in from keytab as oryx/127.28.191.3@KRBTEST.COM (short username oryx)
I20260806 02:40:57.698778 8343 webserver.cc:460] Webserver started at http://127.28.191.3:33581/ using document root <none> and password file <none>
I20260806 02:40:57.699335 8343 fs_manager.cc:359] Metadata directory not provided
I20260806 02:40:57.699425 8343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:40:57.699677 8343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:40:57.701732 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "e961a77795744efa823a5df081a7375b"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "78f69e709c5ab05ed416daa1aad4f571"
server_key_iv: "0d0d0f839ddb111b941830ccc0dca0d9"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.702322 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "e961a77795744efa823a5df081a7375b"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "78f69e709c5ab05ed416daa1aad4f571"
server_key_iv: "0d0d0f839ddb111b941830ccc0dca0d9"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.705943 8343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:40:57.708236 8371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.709430 8343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:40:57.709594 8343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "e961a77795744efa823a5df081a7375b"
format_stamp: "Formatted at 2026-08-06 02:40:57 on dist-test-slave-tfpf"
server_key: "78f69e709c5ab05ed416daa1aad4f571"
server_key_iv: "0d0d0f839ddb111b941830ccc0dca0d9"
server_key_version: "encryptionkey@0"
I20260806 02:40:57.709702 8343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:40:57.727063 8343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:40:57.729831 8343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:40:57.730023 8343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:40:57.731338 8343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:40:57.731422 8343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:40:57.731511 8343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:40:57.731570 8343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:57.743713 8343 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:34623
I20260806 02:40:57.744041 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:34623 every 8 connection(s)
I20260806 02:40:57.744634 8343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:40:57.746162 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 8343
I20260806 02:40:57.746289 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:40:57.746587 29436 external_mini_cluster.cc:1428] Setting key 52dcb45ab6709a74fe3cf08b80fedf5b
I20260806 02:40:57.746748 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.732210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:45457 (local address 127.28.191.62:41343)
0806 02:40:57.732357 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.732361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.733185 (+ 824us) server_negotiation.cc:407] Connection header received
0806 02:40:57.734133 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.734137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.734200 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.734284 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.735966 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.736928 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.738535 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.738820 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.738916 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.742762 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.742765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.742768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.742801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.744529 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.745116 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.745120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.745122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.745210 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.745616 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.745619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.745621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.745814 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.745941 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.746452 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.746576 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:40:57.747660 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.732565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.3:45457)
0806 02:40:57.732997 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.733055 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.733900 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.734611 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.734636 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.735142 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.735791 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.735828 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.737094 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.737099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.738389 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.738403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.739012 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.739018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.739722 (+ 704us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.740883 (+ 1161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.740899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.742599 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.744653 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.744659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.744672 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.744966 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.745319 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.745322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.745324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.745489 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.745941 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.745947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.746181 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.746387 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.746694 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260806 02:40:57.748997 8545 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41343
I20260806 02:40:57.749189 8545 heartbeater.cc:464] Registering TS with master...
I20260806 02:40:57.749728 8545 heartbeater.cc:511] Master 127.28.191.62:41343 requested a full tablet report, sending...
I20260806 02:40:57.750733 7824 ts_manager.cc:194] Registered new tserver with Master: e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:57.751472 7824 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:45457
I20260806 02:40:57.757800 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.751012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.28.191.3:54415)
0806 02:40:57.751286 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.751299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.751386 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.751681 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.751685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.751884 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:57.751978 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.751983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.752721 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.752724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.753333 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.753342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.753813 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.753817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.753911 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.754357 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:57.754371 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:57.754684 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:57.756273 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.756277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.756279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.756470 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.756774 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:57.756776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:57.756779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:57.756857 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:57.757294 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:57.757297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:57.757391 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.757495 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.757628 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260806 02:40:57.757827 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.751080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:54415 (local address 127.28.191.62:41343)
0806 02:40:57.751201 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.751205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.751349 (+ 144us) server_negotiation.cc:407] Connection header received
0806 02:40:57.751460 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.751462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.751496 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.751605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.752103 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.752595 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.753447 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.753699 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.753808 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.754812 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.754816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.754818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.754841 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.756161 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.756570 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.756574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.756576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.756660 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.756948 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.756953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.756954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.757196 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.757304 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.757585 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.757691 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:40:57.760710 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:40:57.785931 7898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:57.779741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.28.191.62:41343)
0806 02:40:57.779868 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.779871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.779884 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:57.779928 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.779930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.779970 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.780047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.780717 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.781290 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.782084 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.782295 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.782373 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.785776 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42142: BlockingRecv error: recv got EOF from 127.0.0.1:42142 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 02:40:57 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:57.795173 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.787244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.28.191.62:41343)
0806 02:40:57.787367 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.787370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.787509 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:40:57.787606 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.787609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.787646 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.787724 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:57.788259 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.788742 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.789521 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.789725 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.789806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.791932 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:57.791938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:57.791943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:57.791978 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:57.793549 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.794012 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.794015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.794016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.794103 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:57.794347 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:57.794349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:57.794350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:57.794509 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:57.794609 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.794855 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.794971 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:40:57.798089 7824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:57.801045 7824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:57.815614 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.810158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.28.191.3:34623)
0806 02:40:57.810654 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.810667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.811223 (+ 556us) server_negotiation.cc:407] Connection header received
0806 02:40:57.811314 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.811319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.811490 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.811619 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:57.811807 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812559 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.814385 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.814922 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815054 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815224 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.815370 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.815457 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":119,"threads_started":1}
I20260806 02:40:57.815603 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.809949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:34623 (local address 127.0.0.1:58510)
0806 02:40:57.811018 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.811033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.811119 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.811604 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.811607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.811623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:57.811703 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.811711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812680 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.812686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.814259 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.814270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.815061 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.815168 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815209 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.815338 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.815457 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":105,"thread_start_us":86,"threads_started":1}
I20260806 02:40:57.816002 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.809644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34327 (local address 127.0.0.1:34564)
0806 02:40:57.810639 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.810653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.810829 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.811318 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.811320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.811338 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:57.811433 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.811438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812710 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.812713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.814101 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.814111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.815432 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.815625 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815714 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.815785 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.815873 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":135,"thread_start_us":109,"threads_started":1}
I20260806 02:40:57.816375 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.809271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37411 (local address 127.0.0.1:50224)
0806 02:40:57.809843 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.809880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.810050 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.810914 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.810921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.810947 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:57.811091 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.811098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812480 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.812483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.814206 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.814219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.815432 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.815633 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815720 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.815785 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.816256 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":632,"thread_start_us":63,"threads_started":1}
I20260806 02:40:57.816576 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.809381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.28.191.1:37411)
0806 02:40:57.809633 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.809643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.810209 (+ 566us) server_negotiation.cc:407] Connection header received
0806 02:40:57.810304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.810310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.810541 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.810716 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:57.811235 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812360 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.814322 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.815092 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815218 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815364 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.815874 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.815963 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260806 02:40:57.816396 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.810142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.28.191.2:34327)
0806 02:40:57.810454 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.810465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.810735 (+ 270us) server_negotiation.cc:407] Connection header received
0806 02:40:57.810973 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.810979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.811182 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.811347 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:57.811528 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.812580 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.814299 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.815124 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.815218 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.815382 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.816183 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.816273 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
I20260806 02:40:57.819094 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5e2526e944fb99f9f0207177195ea (DEFAULT_TABLE table=test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.819242 8229 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5e2526e944fb99f9f0207177195ea (DEFAULT_TABLE table=test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.819778 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5e2526e944fb99f9f0207177195ea (DEFAULT_TABLE table=test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.820195 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5e2526e944fb99f9f0207177195ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.820278 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5e2526e944fb99f9f0207177195ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.820672 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5e2526e944fb99f9f0207177195ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.827008 8592 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Bootstrap starting.
I20260806 02:40:57.828079 8594 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Bootstrap starting.
I20260806 02:40:57.828445 8595 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Bootstrap starting.
I20260806 02:40:57.829288 8592 tablet_bootstrap.cc:654] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.829800 8594 tablet_bootstrap.cc:654] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.830179 8592 log.cc:824] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:57.830436 8595 tablet_bootstrap.cc:654] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.830557 8594 log.cc:824] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:57.831296 8595 log.cc:824] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:40:57.832233 8594 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: No bootstrap required, opened a new log
I20260806 02:40:57.832233 8592 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: No bootstrap required, opened a new log
I20260806 02:40:57.832423 8592 ts_tablet_manager.cc:1397] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:40:57.832423 8594 ts_tablet_manager.cc:1397] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260806 02:40:57.833030 8595 tablet_bootstrap.cc:492] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: No bootstrap required, opened a new log
I20260806 02:40:57.833253 8595 ts_tablet_manager.cc:1397] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260806 02:40:57.836053 8595 raft_consensus.cc:348] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.836285 8595 raft_consensus.cc:374] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.836369 8595 raft_consensus.cc:729] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.836428 8592 raft_consensus.cc:348] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.836732 8592 raft_consensus.cc:374] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.836802 8592 raft_consensus.cc:729] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e961a77795744efa823a5df081a7375b, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.836784 8594 raft_consensus.cc:348] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.836905 8595 consensus_queue.cc:260] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.837109 8594 raft_consensus.cc:374] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.837181 8594 raft_consensus.cc:729] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.837232 8592 consensus_queue.cc:260] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.837594 8594 consensus_queue.cc:260] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.838018 8595 ts_tablet_manager.cc:1428] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260806 02:40:57.838193 8592 ts_tablet_manager.cc:1428] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:40:57.838449 8594 ts_tablet_manager.cc:1428] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 02:40:57.838449 8328 heartbeater.cc:503] Master 127.28.191.62:41343 was elected leader, sending a full tablet report...
I20260806 02:40:57.838488 8545 heartbeater.cc:503] Master 127.28.191.62:41343 was elected leader, sending a full tablet report...
I20260806 02:40:57.838819 8119 heartbeater.cc:503] Master 127.28.191.62:41343 was elected leader, sending a full tablet report...
I20260806 02:40:57.880301 8602 raft_consensus.cc:482] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:57.880479 8602 raft_consensus.cc:504] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.881508 8602 leader_election.cc:288] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:57.885525 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.881808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37411 (local address 127.28.191.2:43395)
0806 02:40:57.881977 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.881994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.882121 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.882422 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.882425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.882442 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:57.882578 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.882584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.883333 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.883338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.884204 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.884216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.885086 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.885090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.885203 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.885221 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.885321 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.885394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:40:57.885536 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.881920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:43395 (local address 127.28.191.1:37411)
0806 02:40:57.882072 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.882078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.882095 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:40:57.882245 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.882250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.882306 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.882390 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:57.882689 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.883217 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.884347 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.884898 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.885019 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.885111 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.885347 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.885416 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:40:57.886200 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5e2526e944fb99f9f0207177195ea" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2" is_pre_election: true
I20260806 02:40:57.886555 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.882098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:34623 (local address 127.28.191.2:56989)
0806 02:40:57.882456 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:57.882470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:57.882621 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:57.882876 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:57.882879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:57.882893 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:57.883021 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.883025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.883735 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.883741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.885310 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:57.885319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.886147 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.886150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:57.886251 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.886271 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:40:57.886361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:40:57.886431 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":148,"threads_started":1}
I20260806 02:40:57.886546 8051 raft_consensus.cc:2454] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:57.886574 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.882324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:56989 (local address 127.28.191.3:34623)
0806 02:40:57.882448 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.882452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.882572 (+ 120us) server_negotiation.cc:407] Connection header received
0806 02:40:57.882727 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.882730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.882774 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.882854 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:57.883126 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.883621 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.885429 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.886015 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.886164 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.886212 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.886409 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.886474 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:40:57.887162 8160 leader_election.cc:302] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:57.887418 8602 raft_consensus.cc:2781] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:57.887524 8602 raft_consensus.cc:482] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:57.887573 8602 raft_consensus.cc:3037] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:57.887838 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5e2526e944fb99f9f0207177195ea" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b" is_pre_election: true
I20260806 02:40:57.888096 8454 raft_consensus.cc:2454] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:57.888852 8602 raft_consensus.cc:504] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.889221 8602 leader_election.cc:288] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Requested vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:57.889588 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5e2526e944fb99f9f0207177195ea" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b"
I20260806 02:40:57.889657 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5e2526e944fb99f9f0207177195ea" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2"
I20260806 02:40:57.889698 8454 raft_consensus.cc:3037] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:57.889801 8051 raft_consensus.cc:3037] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:57.890766 8051 raft_consensus.cc:2454] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:57.890836 8454 raft_consensus.cc:2454] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:57.891095 8160 leader_election.cc:302] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:57.891319 8602 raft_consensus.cc:2781] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:57.891554 8602 raft_consensus.cc:686] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Becoming Leader. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Running, Role: LEADER
I20260806 02:40:57.891868 8602 consensus_queue.cc:237] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } }
I20260806 02:40:57.895215 7824 catalog_manager.cc:5495] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d35be173cd140b1a6df29da1c45155a (127.28.191.2). New cstate: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } health_report { overall_health: HEALTHY } } }
I20260806 02:40:57.911805 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:57.908240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.28.191.2:34327)
0806 02:40:57.908377 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:57.908381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:57.908395 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:57.908574 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:57.908578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:57.908627 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:57.908728 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:57.908981 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.909525 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.910401 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:57.910629 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:57.910735 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:57.911018 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:57.911411 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:57.911536 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:57.911622 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 02:40:57.911674 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:40:57.917403 8454 raft_consensus.cc:1261] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:57.917403 8051 raft_consensus.cc:1261] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:57.918123 8602 consensus_queue.cc:1035] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:57.918298 8607 consensus_queue.cc:1035] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:57.925200 8612 mvcc.cc:205] Tried to move back new op lower bound from 7315390701222449152 to 7315390701127872512. Current Snapshot: MvccSnapshot[applied={T|T < 7315390701222449152}]
I20260806 02:40:57.928604 8613 mvcc.cc:205] Tried to move back new op lower bound from 7315390701222449152 to 7315390701127872512. Current Snapshot: MvccSnapshot[applied={T|T < 7315390701222449152}]
I20260806 02:40:57.933449 7826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:57.933584 7826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:57.935770 7826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Sending DeleteTablet for 3 replicas of tablet 69e5e2526e944fb99f9f0207177195ea
I20260806 02:40:57.936537 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5e2526e944fb99f9f0207177195ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:57 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260806 02:40:57.936635 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5e2526e944fb99f9f0207177195ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:57 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:58510
I20260806 02:40:57.937402 7826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:40:57.937561 8625 tablet_replica.cc:330] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: stopping tablet replica
I20260806 02:40:57.937578 8229 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5e2526e944fb99f9f0207177195ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:57 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260806 02:40:57.937858 8625 raft_consensus.cc:2229] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 02:40:57.938047 7826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:57.938060 8626 tablet_replica.cc:330] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: stopping tablet replica
I20260806 02:40:57.938143 8625 raft_consensus.cc:2258] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:57.938313 8626 raft_consensus.cc:2229] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:57.938516 8624 tablet_replica.cc:330] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: stopping tablet replica
I20260806 02:40:57.938683 8626 raft_consensus.cc:2258] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:57.938817 8624 raft_consensus.cc:2229] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:57.939118 8624 raft_consensus.cc:2258] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:57.940261 8626 ts_tablet_manager.cc:1905] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:57.940408 8624 ts_tablet_manager.cc:1905] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:57.943190 8624 ts_tablet_manager.cc:1918] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:57.943332 8624 log.cc:1196] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/69e5e2526e944fb99f9f0207177195ea
I20260806 02:40:57.943701 8624 ts_tablet_manager.cc:1939] T 69e5e2526e944fb99f9f0207177195ea P 64145fe1927c47528bf330fda7f444b2: Deleting consensus metadata
I20260806 02:40:57.943691 8229 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 (DEFAULT_TABLE table=test-table [id=4d68426472a1465ca4068a5e7914704f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.943727 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 (DEFAULT_TABLE table=test-table [id=4d68426472a1465ca4068a5e7914704f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.943957 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7310abab5d4ec8a6055021bed99d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.943966 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7310abab5d4ec8a6055021bed99d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.946003 8594 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Bootstrap starting.
I20260806 02:40:57.946074 7809 catalog_manager.cc:4841] TS 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411): tablet 69e5e2526e944fb99f9f0207177195ea (table test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]) successfully deleted
I20260806 02:40:57.946393 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 (DEFAULT_TABLE table=test-table [id=4d68426472a1465ca4068a5e7914704f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:57.946657 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a7310abab5d4ec8a6055021bed99d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:57.946935 8594 tablet_bootstrap.cc:654] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.947708 8625 ts_tablet_manager.cc:1905] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:57.949064 8592 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Bootstrap starting.
I20260806 02:40:57.949299 8595 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Bootstrap starting.
I20260806 02:40:57.949810 8592 tablet_bootstrap.cc:654] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.950199 8595 tablet_bootstrap.cc:654] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:57.951017 8592 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: No bootstrap required, opened a new log
I20260806 02:40:57.951126 8592 ts_tablet_manager.cc:1397] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:40:57.951567 8595 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: No bootstrap required, opened a new log
I20260806 02:40:57.951639 8595 ts_tablet_manager.cc:1397] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 02:40:57.951581 8592 raft_consensus.cc:348] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.951696 8592 raft_consensus.cc:374] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.951731 8592 raft_consensus.cc:729] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e961a77795744efa823a5df081a7375b, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.951865 8592 consensus_queue.cc:260] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.952046 8595 raft_consensus.cc:348] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.952112 8592 ts_tablet_manager.cc:1428] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:57.952142 8595 raft_consensus.cc:374] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.952172 8595 raft_consensus.cc:729] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.952301 8595 consensus_queue.cc:260] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.952548 8595 ts_tablet_manager.cc:1428] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:40:57.953552 8625 ts_tablet_manager.cc:1918] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:57.943446 8626 ts_tablet_manager.cc:1918] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:57.953838 8625 log.cc:1196] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/69e5e2526e944fb99f9f0207177195ea
I20260806 02:40:57.954030 8626 log.cc:1196] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/69e5e2526e944fb99f9f0207177195ea
I20260806 02:40:57.954265 8625 ts_tablet_manager.cc:1939] T 69e5e2526e944fb99f9f0207177195ea P e961a77795744efa823a5df081a7375b: Deleting consensus metadata
I20260806 02:40:57.954385 8626 ts_tablet_manager.cc:1939] T 69e5e2526e944fb99f9f0207177195ea P 8d35be173cd140b1a6df29da1c45155a: Deleting consensus metadata
I20260806 02:40:57.955274 7809 catalog_manager.cc:4841] TS e961a77795744efa823a5df081a7375b (127.28.191.3:34623): tablet 69e5e2526e944fb99f9f0207177195ea (table test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]) successfully deleted
I20260806 02:40:57.955536 7804 catalog_manager.cc:4841] TS 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327): tablet 69e5e2526e944fb99f9f0207177195ea (table test-table [id=f2435d58e3404fb4be9c7c9c726df9ca]) successfully deleted
I20260806 02:40:57.957805 8594 tablet_bootstrap.cc:492] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: No bootstrap required, opened a new log
I20260806 02:40:57.957906 8594 ts_tablet_manager.cc:1397] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260806 02:40:57.958714 8594 raft_consensus.cc:348] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.958818 8594 raft_consensus.cc:374] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:57.958850 8594 raft_consensus.cc:729] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Initialized, Role: FOLLOWER
I20260806 02:40:57.958964 8594 consensus_queue.cc:260] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:57.959195 8594 ts_tablet_manager.cc:1428] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260806 02:40:57.996126 8547 tablet.cc:2369] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:40:58.021248 8331 tablet.cc:2369] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:40:58.029052 8607 raft_consensus.cc:482] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:58.029217 8607 raft_consensus.cc:504] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.029640 8607 leader_election.cc:288] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.030140 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7310abab5d4ec8a6055021bed99d38" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b" is_pre_election: true
I20260806 02:40:58.030172 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7310abab5d4ec8a6055021bed99d38" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2" is_pre_election: true
I20260806 02:40:58.030290 8454 raft_consensus.cc:2454] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.030321 8051 raft_consensus.cc:2454] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.030596 8160 leader_election.cc:302] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.030778 8607 raft_consensus.cc:2781] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:58.030866 8607 raft_consensus.cc:482] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:58.030912 8607 raft_consensus.cc:3037] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.031917 8607 raft_consensus.cc:504] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.032274 8607 leader_election.cc:288] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Requested vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.032692 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7310abab5d4ec8a6055021bed99d38" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2"
I20260806 02:40:58.032750 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a7310abab5d4ec8a6055021bed99d38" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b"
I20260806 02:40:58.032811 8051 raft_consensus.cc:3037] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.032868 8454 raft_consensus.cc:3037] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.034030 8051 raft_consensus.cc:2454] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.034190 8454 raft_consensus.cc:2454] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.034432 8160 leader_election.cc:302] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.034665 8607 raft_consensus.cc:2781] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:58.034785 8607 raft_consensus.cc:686] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Becoming Leader. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Running, Role: LEADER
I20260806 02:40:58.034951 8607 consensus_queue.cc:237] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.036733 7824 catalog_manager.cc:5495] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d35be173cd140b1a6df29da1c45155a (127.28.191.2). New cstate: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } health_report { overall_health: UNKNOWN } } }
W20260806 02:40:58.040614 8120 tablet.cc:2369] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.62@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:58.088241 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.078794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41343 (local address 127.0.0.1:42162)
0806 02:40:58.079143 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.079160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:58.079285 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:58.079559 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:58.079562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:58.080064 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:40:58.080184 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.080192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.080910 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.080914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.081878 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.081890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.082331 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.082335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.082421 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.083091 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:40:58.083107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:40:58.084741 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:40:58.086606 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:58.086617 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:58.086623 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:58.086845 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:58.087193 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:40:58.087196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:40:58.087198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:40:58.087286 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:40:58.087658 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:40:58.087666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:40:58.087775 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:40:58.087889 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:40:58.088052 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260806 02:40:58.088240 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.078926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.28.191.62:41343)
0806 02:40:58.079059 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.079064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.079246 (+ 182us) server_negotiation.cc:407] Connection header received
0806 02:40:58.079373 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.079375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.079415 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.079483 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:58.080304 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.080797 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.082006 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.082219 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.082296 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.084881 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:58.084884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:58.084886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:58.084905 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:58.086503 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:58.086943 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:58.086948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:58.086950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:58.087085 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:58.087381 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:58.087384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:58.087386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:58.087554 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:58.087658 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.087946 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.088106 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":50}
I20260806 02:40:58.092131 7824 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:42162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:40:58.092890 7824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:58.097777 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 98e507f2e0a14467a46a7f175e918017 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27386d6504d2454d8b2ef6bedbbba919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:58.097831 8229 tablet_service.cc:1467] Processing CreateTablet for tablet 98e507f2e0a14467a46a7f175e918017 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27386d6504d2454d8b2ef6bedbbba919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:58.097937 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 98e507f2e0a14467a46a7f175e918017 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=27386d6504d2454d8b2ef6bedbbba919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:40:58.098188 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e507f2e0a14467a46a7f175e918017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.098229 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e507f2e0a14467a46a7f175e918017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.098261 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e507f2e0a14467a46a7f175e918017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.100505 8594 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: Bootstrap starting.
I20260806 02:40:58.100857 8595 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: Bootstrap starting.
I20260806 02:40:58.101464 8594 tablet_bootstrap.cc:654] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.101660 8595 tablet_bootstrap.cc:654] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.102411 8592 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: Bootstrap starting.
I20260806 02:40:58.102771 8594 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: No bootstrap required, opened a new log
I20260806 02:40:58.102840 8594 ts_tablet_manager.cc:1397] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 02:40:58.103278 8594 raft_consensus.cc:348] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.103389 8592 tablet_bootstrap.cc:654] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.103431 8594 raft_consensus.cc:374] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.103459 8594 raft_consensus.cc:729] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.103624 8594 consensus_queue.cc:260] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.103942 8594 ts_tablet_manager.cc:1428] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:40:58.103885 8605 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.104025 8605 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.104351 8595 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: No bootstrap required, opened a new log
I20260806 02:40:58.104468 8595 ts_tablet_manager.cc:1397] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 02:40:58.104803 8595 raft_consensus.cc:348] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.104918 8595 raft_consensus.cc:374] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.104964 8595 raft_consensus.cc:729] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.105110 8595 consensus_queue.cc:260] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.105400 8595 ts_tablet_manager.cc:1428] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.105397 8607 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.106122 8607 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.106236 8592 tablet_bootstrap.cc:492] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: No bootstrap required, opened a new log
I20260806 02:40:58.106352 8592 ts_tablet_manager.cc:1397] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:40:58.106794 8592 raft_consensus.cc:348] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.106941 8592 raft_consensus.cc:374] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.106987 8592 raft_consensus.cc:729] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e961a77795744efa823a5df081a7375b, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.107137 8592 consensus_queue.cc:260] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.107425 8592 ts_tablet_manager.cc:1428] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.107441 8603 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.107998 8603 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.161398 8607 raft_consensus.cc:482] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:58.161522 8607 raft_consensus.cc:504] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.161859 8607 leader_election.cc:288] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.162304 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e507f2e0a14467a46a7f175e918017" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2" is_pre_election: true
I20260806 02:40:58.162472 8051 raft_consensus.cc:2454] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.162408 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e507f2e0a14467a46a7f175e918017" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b" is_pre_election: true
I20260806 02:40:58.162614 8454 raft_consensus.cc:2454] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.162779 8160 leader_election.cc:302] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.162957 8607 raft_consensus.cc:2781] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:58.163033 8607 raft_consensus.cc:482] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:58.163087 8607 raft_consensus.cc:3037] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.164052 8607 raft_consensus.cc:504] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.164397 8607 leader_election.cc:288] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Requested vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.164723 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e507f2e0a14467a46a7f175e918017" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2"
I20260806 02:40:58.164767 8454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e507f2e0a14467a46a7f175e918017" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b"
I20260806 02:40:58.164848 8051 raft_consensus.cc:3037] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.164888 8454 raft_consensus.cc:3037] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.165987 8051 raft_consensus.cc:2454] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.165997 8454 raft_consensus.cc:2454] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.166283 8160 leader_election.cc:302] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.166481 8607 raft_consensus.cc:2781] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:58.166590 8607 raft_consensus.cc:686] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Becoming Leader. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Running, Role: LEADER
I20260806 02:40:58.166739 8607 consensus_queue.cc:237] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.167480 8602 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: TxnStatusTablet state changed. Reason: New leader 8d35be173cd140b1a6df29da1c45155a. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.167586 8602 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:58.167882 8651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:40:58.168084 8651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:40:58.168445 7826 catalog_manager.cc:5495] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d35be173cd140b1a6df29da1c45155a (127.28.191.2). New cstate: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:58.189620 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.186157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34327 (local address 127.0.0.1:34578)
0806 02:40:58.186476 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.186491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:58.186626 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:58.186913 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:58.186916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:58.186924 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:58.187029 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.187053 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.187758 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.187761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.188350 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.188359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.188831 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.188834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.188925 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.188978 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.189335 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.189353 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:40:58.189418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:40:58.189497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260806 02:40:58.189620 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.186280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.28.191.2:34327)
0806 02:40:58.186428 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.186434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.186565 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:40:58.186750 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.186756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.186810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.186889 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:58.187151 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.187647 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.188494 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.188717 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.188830 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.189170 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.189247 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.189330 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.189429 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.189478 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:40:58.191912 8051 raft_consensus.cc:1261] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.191982 8454 raft_consensus.cc:1261] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.192386 8607 consensus_queue.cc:1035] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.192557 8602 consensus_queue.cc:1035] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.193924 8605 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: TxnStatusTablet state changed. Reason: New leader 8d35be173cd140b1a6df29da1c45155a. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.193944 8603 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: TxnStatusTablet state changed. Reason: New leader 8d35be173cd140b1a6df29da1c45155a. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.194036 8605 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.194074 8603 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.196626 8602 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.196744 8602 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:58.196892 8602 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.196971 8602 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 8d35be173cd140b1a6df29da1c45155a: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:40:58.199172 8605 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.199287 8605 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P 64145fe1927c47528bf330fda7f444b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.202620 8603 tablet_replica.cc:439] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } } }
I20260806 02:40:58.202762 8603 tablet_replica.cc:442] T 98e507f2e0a14467a46a7f175e918017 P e961a77795744efa823a5df081a7375b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:40:58.211727 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.207954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:44859 (local address 127.28.191.2:34327)
0806 02:40:58.208059 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.208063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.208077 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:40:58.208204 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.208206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.208241 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.208318 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:58.208591 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.209033 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.210452 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.210832 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.210924 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.211297 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.211364 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.211439 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.211569 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.211609 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:40:58.211835 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.207688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34327 (local address 127.28.191.2:44859)
0806 02:40:58.208003 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.208017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:58.208108 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:58.208359 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:58.208362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:58.208370 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:40:58.208484 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.208490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.209151 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.209154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.210173 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.210203 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.210960 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.210964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.211054 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.211107 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.211472 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.211478 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:40:58.211554 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:40:58.211636 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260806 02:40:58.218360 8454 raft_consensus.cc:1261] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.218369 8051 raft_consensus.cc:1261] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.218724 8660 consensus_queue.cc:1035] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.218905 8602 consensus_queue.cc:1035] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.226617 8614 mvcc.cc:205] Tried to move back new op lower bound from 7315390702459260928 to 7315390701712932864. Current Snapshot: MvccSnapshot[applied={T|T < 7315390702459260928}]
I20260806 02:40:58.257839 7821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.258008 7821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.260053 7821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Sending DeleteTablet for 3 replicas of tablet 8a7310abab5d4ec8a6055021bed99d38
I20260806 02:40:58.260602 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260806 02:40:58.260756 8229 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260806 02:40:58.260789 8624 tablet_replica.cc:330] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: stopping tablet replica
I20260806 02:40:58.260896 8624 raft_consensus.cc:2229] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.260933 8626 tablet_replica.cc:330] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: stopping tablet replica
I20260806 02:40:58.261009 8624 raft_consensus.cc:2258] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.261065 8626 raft_consensus.cc:2229] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:58.261240 8626 raft_consensus.cc:2258] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.261776 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a7310abab5d4ec8a6055021bed99d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:58510
I20260806 02:40:58.261888 8626 ts_tablet_manager.cc:1905] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.262109 8624 ts_tablet_manager.cc:1905] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.264098 8624 ts_tablet_manager.cc:1918] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:58.264182 8624 log.cc:1196] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/8a7310abab5d4ec8a6055021bed99d38
I20260806 02:40:58.264370 8624 ts_tablet_manager.cc:1939] T 8a7310abab5d4ec8a6055021bed99d38 P 64145fe1927c47528bf330fda7f444b2: Deleting consensus metadata
I20260806 02:40:58.264572 8626 ts_tablet_manager.cc:1918] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:58.264639 8626 log.cc:1196] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/8a7310abab5d4ec8a6055021bed99d38
I20260806 02:40:58.264878 8626 ts_tablet_manager.cc:1939] T 8a7310abab5d4ec8a6055021bed99d38 P 8d35be173cd140b1a6df29da1c45155a: Deleting consensus metadata
I20260806 02:40:58.264914 8625 tablet_replica.cc:330] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: stopping tablet replica
I20260806 02:40:58.264961 7809 catalog_manager.cc:4841] TS 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411): tablet 8a7310abab5d4ec8a6055021bed99d38 (table test-table [id=4d68426472a1465ca4068a5e7914704f]) successfully deleted
I20260806 02:40:58.265035 8625 raft_consensus.cc:2229] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.265231 8625 raft_consensus.cc:2258] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.265687 7804 catalog_manager.cc:4841] TS 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327): tablet 8a7310abab5d4ec8a6055021bed99d38 (table test-table [id=4d68426472a1465ca4068a5e7914704f]) successfully deleted
I20260806 02:40:58.269625 8625 ts_tablet_manager.cc:1905] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.271853 8625 ts_tablet_manager.cc:1918] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:58.271936 8625 log.cc:1196] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/8a7310abab5d4ec8a6055021bed99d38
I20260806 02:40:58.272203 8625 ts_tablet_manager.cc:1939] T 8a7310abab5d4ec8a6055021bed99d38 P e961a77795744efa823a5df081a7375b: Deleting consensus metadata
I20260806 02:40:58.272873 7809 catalog_manager.cc:4841] TS e961a77795744efa823a5df081a7375b (127.28.191.3:34623): tablet 8a7310abab5d4ec8a6055021bed99d38 (table test-table [id=4d68426472a1465ca4068a5e7914704f]) successfully deleted
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[7760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:40:58.295682 7898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:40:58.288445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.28.191.62:41343)
0806 02:40:58.288590 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.288595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.288828 (+ 233us) server_negotiation.cc:407] Connection header received
0806 02:40:58.288987 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.288991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.289049 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.289194 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:58.290072 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.290606 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.291641 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.291911 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.291974 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.295557 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42176: BlockingRecv error: recv got EOF from 127.0.0.1:42176 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[7760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM
I20260806 02:40:58.305728 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.297592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.28.191.62:41343)
0806 02:40:58.297699 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.297703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.297748 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:40:58.297907 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.297911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.297958 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.298047 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:40:58.298624 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.299127 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.299930 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.300145 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.300247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.302520 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:40:58.302523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:40:58.302525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:40:58.302552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:40:58.303966 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:58.304394 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:58.304398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:58.304400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:58.304480 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:40:58.304780 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:40:58.304784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:40:58.304785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:40:58.304962 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:40:58.305175 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.305410 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.305526 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:40:58.308272 7821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:58.308743 7821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:58.313720 8023 tablet_service.cc:1467] Processing CreateTablet for tablet c015c68e53e848a89dacd14a3d89293f (DEFAULT_TABLE table=test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.313808 8229 tablet_service.cc:1467] Processing CreateTablet for tablet c015c68e53e848a89dacd14a3d89293f (DEFAULT_TABLE table=test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.314023 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c015c68e53e848a89dacd14a3d89293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.314086 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c015c68e53e848a89dacd14a3d89293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.314343 8436 tablet_service.cc:1467] Processing CreateTablet for tablet c015c68e53e848a89dacd14a3d89293f (DEFAULT_TABLE table=test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.314632 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c015c68e53e848a89dacd14a3d89293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.316486 8595 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Bootstrap starting.
I20260806 02:40:58.316596 8594 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Bootstrap starting.
I20260806 02:40:58.317129 8592 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Bootstrap starting.
I20260806 02:40:58.317389 8594 tablet_bootstrap.cc:654] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.317387 8595 tablet_bootstrap.cc:654] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.317996 8592 tablet_bootstrap.cc:654] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.319218 8594 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: No bootstrap required, opened a new log
I20260806 02:40:58.319306 8594 ts_tablet_manager.cc:1397] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:40:58.319574 8592 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: No bootstrap required, opened a new log
I20260806 02:40:58.319581 8595 tablet_bootstrap.cc:492] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: No bootstrap required, opened a new log
I20260806 02:40:58.319646 8592 ts_tablet_manager.cc:1397] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:40:58.319662 8595 ts_tablet_manager.cc:1397] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:40:58.319895 8594 raft_consensus.cc:348] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.319998 8594 raft_consensus.cc:374] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.320031 8594 raft_consensus.cc:729] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.320019 8595 raft_consensus.cc:348] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.320060 8592 raft_consensus.cc:348] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.320125 8595 raft_consensus.cc:374] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.320154 8595 raft_consensus.cc:729] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.320158 8592 raft_consensus.cc:374] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.320199 8592 raft_consensus.cc:729] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e961a77795744efa823a5df081a7375b, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.320178 8594 consensus_queue.cc:260] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.320279 8595 consensus_queue.cc:260] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.320319 8592 consensus_queue.cc:260] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.320451 8594 ts_tablet_manager.cc:1428] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.320596 8592 ts_tablet_manager.cc:1428] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.320607 8595 ts_tablet_manager.cc:1428] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.441301 8605 raft_consensus.cc:482] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:58.441473 8605 raft_consensus.cc:504] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.442780 8605 leader_election.cc:288] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.446635 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.443076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:59935 (local address 127.28.191.2:34327)
0806 02:40:58.443227 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.443232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.443291 (+ 59us) server_negotiation.cc:407] Connection header received
0806 02:40:58.443477 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.443481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.443537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.443612 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:58.443869 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.444440 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.445643 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.446058 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.446188 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.446243 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.446446 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.446517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:40:58.446671 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.442938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:34327 (local address 127.28.191.1:59935)
0806 02:40:58.443193 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.443209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:58.443336 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:58.443617 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:58.443619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:58.443633 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:58.443758 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.443763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.444560 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.444565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.445524 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.445534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.446181 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.446185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.446291 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.446311 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:40:58.446428 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:40:58.446510 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260806 02:40:58.446890 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.443071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:34623 (local address 127.28.191.1:35485)
0806 02:40:58.443382 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.443399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:40:58.443496 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:40:58.443904 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:40:58.443907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:40:58.443923 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:40:58.444058 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.444064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.444835 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.444838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.445704 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:40:58.445713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.446428 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.446432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:40:58.446560 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.446586 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:40:58.446674 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:40:58.446752 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":27,"thread_start_us":74,"threads_started":1}
I20260806 02:40:58.446887 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.443452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:35485 (local address 127.28.191.3:34623)
0806 02:40:58.443700 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.443703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.443714 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:40:58.443758 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.443760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.443795 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.443895 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:40:58.444195 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.444703 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.445828 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.446261 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.446428 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.446478 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.446692 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.446777 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260806 02:40:58.447304 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c015c68e53e848a89dacd14a3d89293f" candidate_uuid: "64145fe1927c47528bf330fda7f444b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d35be173cd140b1a6df29da1c45155a" is_pre_election: true
I20260806 02:40:58.447445 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c015c68e53e848a89dacd14a3d89293f" candidate_uuid: "64145fe1927c47528bf330fda7f444b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b" is_pre_election: true
I20260806 02:40:58.447577 8469 raft_consensus.cc:2454] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64145fe1927c47528bf330fda7f444b2 in term 0.
I20260806 02:40:58.447595 8266 raft_consensus.cc:2454] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64145fe1927c47528bf330fda7f444b2 in term 0.
I20260806 02:40:58.447962 7953 leader_election.cc:302] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, e961a77795744efa823a5df081a7375b; no voters:
I20260806 02:40:58.448233 8605 raft_consensus.cc:2781] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:58.448297 8605 raft_consensus.cc:482] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:58.448366 8605 raft_consensus.cc:3037] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.449523 8605 raft_consensus.cc:504] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.449898 8605 leader_election.cc:288] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.450233 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c015c68e53e848a89dacd14a3d89293f" candidate_uuid: "64145fe1927c47528bf330fda7f444b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d35be173cd140b1a6df29da1c45155a"
I20260806 02:40:58.450294 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c015c68e53e848a89dacd14a3d89293f" candidate_uuid: "64145fe1927c47528bf330fda7f444b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b"
I20260806 02:40:58.450356 8266 raft_consensus.cc:3037] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.450403 8469 raft_consensus.cc:3037] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.451551 8469 raft_consensus.cc:2454] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64145fe1927c47528bf330fda7f444b2 in term 1.
I20260806 02:40:58.451550 8266 raft_consensus.cc:2454] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64145fe1927c47528bf330fda7f444b2 in term 1.
I20260806 02:40:58.451823 7950 leader_election.cc:302] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.452023 8605 raft_consensus.cc:2781] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:58.452194 8605 raft_consensus.cc:686] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 1 LEADER]: Becoming Leader. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Running, Role: LEADER
I20260806 02:40:58.452426 8605 consensus_queue.cc:237] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.454751 7824 catalog_manager.cc:5495] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64145fe1927c47528bf330fda7f444b2 (127.28.191.1). New cstate: current_term: 1 leader_uuid: "64145fe1927c47528bf330fda7f444b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:58.475494 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.471771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.28.191.1:37411)
0806 02:40:58.472029 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.472035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.472047 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:40:58.472117 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.472119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.472187 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.472292 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:58.472512 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.473014 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.473987 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.474241 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.474322 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.474601 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.475061 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.475150 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.475243 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.475290 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260806 02:40:58.478024 8469 raft_consensus.cc:1261] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Refusing update from remote peer 64145fe1927c47528bf330fda7f444b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.478191 8266 raft_consensus.cc:1261] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Refusing update from remote peer 64145fe1927c47528bf330fda7f444b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.478631 8605 consensus_queue.cc:1035] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.478852 8708 consensus_queue.cc:1035] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.490002 7824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.490144 7824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.492321 7824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Sending DeleteTablet for 3 replicas of tablet c015c68e53e848a89dacd14a3d89293f
I20260806 02:40:58.492960 8229 tablet_service.cc:1514] Processing DeleteTablet for tablet c015c68e53e848a89dacd14a3d89293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260806 02:40:58.492990 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet c015c68e53e848a89dacd14a3d89293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260806 02:40:58.493126 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet c015c68e53e848a89dacd14a3d89293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:58510
I20260806 02:40:58.493201 8624 tablet_replica.cc:330] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: stopping tablet replica
I20260806 02:40:58.493216 8626 tablet_replica.cc:330] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: stopping tablet replica
I20260806 02:40:58.493310 8625 tablet_replica.cc:330] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: stopping tablet replica
I20260806 02:40:58.493320 8624 raft_consensus.cc:2229] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:58.493364 8626 raft_consensus.cc:2229] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.493419 8625 raft_consensus.cc:2229] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.493494 8626 raft_consensus.cc:2258] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.493547 8625 raft_consensus.cc:2258] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.493614 8624 raft_consensus.cc:2258] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.494333 8625 ts_tablet_manager.cc:1905] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.494552 8624 ts_tablet_manager.cc:1905] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.496978 8625 ts_tablet_manager.cc:1918] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:58.497032 8624 ts_tablet_manager.cc:1918] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:58.497128 8624 log.cc:1196] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/c015c68e53e848a89dacd14a3d89293f
I20260806 02:40:58.497129 8625 log.cc:1196] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/c015c68e53e848a89dacd14a3d89293f
I20260806 02:40:58.497447 8625 ts_tablet_manager.cc:1939] T c015c68e53e848a89dacd14a3d89293f P e961a77795744efa823a5df081a7375b: Deleting consensus metadata
I20260806 02:40:58.497473 8624 ts_tablet_manager.cc:1939] T c015c68e53e848a89dacd14a3d89293f P 64145fe1927c47528bf330fda7f444b2: Deleting consensus metadata
I20260806 02:40:58.498198 7809 catalog_manager.cc:4841] TS 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411): tablet c015c68e53e848a89dacd14a3d89293f (table test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]) successfully deleted
I20260806 02:40:58.498373 7809 catalog_manager.cc:4841] TS e961a77795744efa823a5df081a7375b (127.28.191.3:34623): tablet c015c68e53e848a89dacd14a3d89293f (table test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]) successfully deleted
I20260806 02:40:58.499259 8626 ts_tablet_manager.cc:1905] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.499563 7824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:40:58.500152 7824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:40:58.501843 8626 ts_tablet_manager.cc:1918] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:40:58.501933 8626 log.cc:1196] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/c015c68e53e848a89dacd14a3d89293f
I20260806 02:40:58.502243 8626 ts_tablet_manager.cc:1939] T c015c68e53e848a89dacd14a3d89293f P 8d35be173cd140b1a6df29da1c45155a: Deleting consensus metadata
I20260806 02:40:58.502943 7804 catalog_manager.cc:4841] TS 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327): tablet c015c68e53e848a89dacd14a3d89293f (table test-table [id=3886b7f811a94ea0a76ea95d8c0eb4e4]) successfully deleted
I20260806 02:40:58.505285 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 15371dc844d242129a09a3433bf057a7 (DEFAULT_TABLE table=test-table [id=838a55b0e194411a9c733acd8330e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.505554 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15371dc844d242129a09a3433bf057a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.505574 8229 tablet_service.cc:1467] Processing CreateTablet for tablet 15371dc844d242129a09a3433bf057a7 (DEFAULT_TABLE table=test-table [id=838a55b0e194411a9c733acd8330e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.505890 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15371dc844d242129a09a3433bf057a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.507689 8595 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Bootstrap starting.
I20260806 02:40:58.507754 8594 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: Bootstrap starting.
I20260806 02:40:58.508641 8594 tablet_bootstrap.cc:654] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.508654 8595 tablet_bootstrap.cc:654] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.508652 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 15371dc844d242129a09a3433bf057a7 (DEFAULT_TABLE table=test-table [id=838a55b0e194411a9c733acd8330e010]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:40:58.509006 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15371dc844d242129a09a3433bf057a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:40:58.510030 8595 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: No bootstrap required, opened a new log
I20260806 02:40:58.510135 8595 ts_tablet_manager.cc:1397] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:40:58.510650 8595 raft_consensus.cc:348] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.510775 8595 raft_consensus.cc:374] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.510835 8595 raft_consensus.cc:729] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.510982 8595 consensus_queue.cc:260] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.511313 8595 ts_tablet_manager.cc:1428] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:40:58.511637 8592 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Bootstrap starting.
I20260806 02:40:58.512526 8592 tablet_bootstrap.cc:654] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Neither blocks nor log segments found. Creating new log.
I20260806 02:40:58.513832 8594 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: No bootstrap required, opened a new log
I20260806 02:40:58.513921 8594 ts_tablet_manager.cc:1397] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:40:58.514320 8594 raft_consensus.cc:348] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.514418 8594 raft_consensus.cc:374] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.514448 8594 raft_consensus.cc:729] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64145fe1927c47528bf330fda7f444b2, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.514576 8594 consensus_queue.cc:260] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.514819 8594 ts_tablet_manager.cc:1428] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.515651 8592 tablet_bootstrap.cc:492] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: No bootstrap required, opened a new log
I20260806 02:40:58.515733 8592 ts_tablet_manager.cc:1397] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:40:58.516131 8592 raft_consensus.cc:348] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.516232 8592 raft_consensus.cc:374] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:40:58.516263 8592 raft_consensus.cc:729] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e961a77795744efa823a5df081a7375b, State: Initialized, Role: FOLLOWER
I20260806 02:40:58.516371 8592 consensus_queue.cc:260] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.516767 8592 ts_tablet_manager.cc:1428] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:40:58.782896 8660 raft_consensus.cc:482] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:40:58.783020 8660 raft_consensus.cc:504] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.783353 8660 leader_election.cc:288] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.783802 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15371dc844d242129a09a3433bf057a7" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2" is_pre_election: true
I20260806 02:40:58.783815 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15371dc844d242129a09a3433bf057a7" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b" is_pre_election: true
I20260806 02:40:58.783946 8469 raft_consensus.cc:2454] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.783999 8051 raft_consensus.cc:2454] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 0.
I20260806 02:40:58.784320 8160 leader_election.cc:302] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d35be173cd140b1a6df29da1c45155a, e961a77795744efa823a5df081a7375b; no voters:
I20260806 02:40:58.784516 8660 raft_consensus.cc:2781] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:40:58.784600 8660 raft_consensus.cc:482] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:40:58.784660 8660 raft_consensus.cc:3037] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.785768 8660 raft_consensus.cc:504] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.786114 8660 leader_election.cc:288] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Requested vote from peers 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411), e961a77795744efa823a5df081a7375b (127.28.191.3:34623)
I20260806 02:40:58.786456 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15371dc844d242129a09a3433bf057a7" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64145fe1927c47528bf330fda7f444b2"
I20260806 02:40:58.786491 8469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15371dc844d242129a09a3433bf057a7" candidate_uuid: "8d35be173cd140b1a6df29da1c45155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e961a77795744efa823a5df081a7375b"
I20260806 02:40:58.786564 8051 raft_consensus.cc:3037] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.786638 8469 raft_consensus.cc:3037] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:40:58.787562 8469 raft_consensus.cc:2454] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.787570 8051 raft_consensus.cc:2454] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d35be173cd140b1a6df29da1c45155a in term 1.
I20260806 02:40:58.787875 8160 leader_election.cc:302] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64145fe1927c47528bf330fda7f444b2, 8d35be173cd140b1a6df29da1c45155a; no voters:
I20260806 02:40:58.788064 8660 raft_consensus.cc:2781] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:40:58.788255 8660 raft_consensus.cc:686] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Becoming Leader. State: Replica: 8d35be173cd140b1a6df29da1c45155a, State: Running, Role: LEADER
I20260806 02:40:58.788417 8660 consensus_queue.cc:237] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } }
I20260806 02:40:58.790148 7824 catalog_manager.cc:5495] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d35be173cd140b1a6df29da1c45155a (127.28.191.2). New cstate: current_term: 1 leader_uuid: "8d35be173cd140b1a6df29da1c45155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d35be173cd140b1a6df29da1c45155a" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 } health_report { overall_health: UNKNOWN } } }
I20260806 02:40:58.816753 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.813979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.28.191.2:34327)
0806 02:40:58.814113 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:40:58.814119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:40:58.814132 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:40:58.814255 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:40:58.814258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:40:58.814297 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:40:58.814355 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:40:58.814598 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.815093 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.815855 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:40:58.816056 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:40:58.816130 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:40:58.816333 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:40:58.816398 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:40:58.816452 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:40:58.816534 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:40:58.816574 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:40:58.822549 8469 raft_consensus.cc:1261] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.822577 8051 raft_consensus.cc:1261] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Refusing update from remote peer 8d35be173cd140b1a6df29da1c45155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:40:58.822913 8724 consensus_queue.cc:1035] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64145fe1927c47528bf330fda7f444b2" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.823096 8660 consensus_queue.cc:1035] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e961a77795744efa823a5df081a7375b" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:40:58.854821 7821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.855003 7821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:40:58.857198 7821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bac4e6f3a0d49e99c8068308a9bbb68: Sending DeleteTablet for 3 replicas of tablet 15371dc844d242129a09a3433bf057a7
I20260806 02:40:58.857728 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 15371dc844d242129a09a3433bf057a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260806 02:40:58.857812 8229 tablet_service.cc:1514] Processing DeleteTablet for tablet 15371dc844d242129a09a3433bf057a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260806 02:40:58.857933 8624 tablet_replica.cc:330] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: stopping tablet replica
I20260806 02:40:58.858017 8626 tablet_replica.cc:330] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: stopping tablet replica
I20260806 02:40:58.858053 8624 raft_consensus.cc:2229] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.858135 8626 raft_consensus.cc:2229] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:40:58.858165 8624 raft_consensus.cc:2258] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.858191 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 15371dc844d242129a09a3433bf057a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:40:58 UTC) from {username='oryx', principal='oryx/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:58510
I20260806 02:40:58.858326 8626 raft_consensus.cc:2258] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.858345 8625 tablet_replica.cc:330] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: stopping tablet replica
I20260806 02:40:58.858448 8625 raft_consensus.cc:2229] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:40:58.858549 8625 raft_consensus.cc:2258] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:40:58.858650 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 7930
I20260806 02:40:58.858870 8624 ts_tablet_manager.cc:1905] T 15371dc844d242129a09a3433bf057a7 P 64145fe1927c47528bf330fda7f444b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.859073 8626 ts_tablet_manager.cc:1905] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.862418 8626 ts_tablet_manager.cc:1918] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:58.862483 8626 log.cc:1196] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/15371dc844d242129a09a3433bf057a7
I20260806 02:40:58.862737 8626 ts_tablet_manager.cc:1939] T 15371dc844d242129a09a3433bf057a7 P 8d35be173cd140b1a6df29da1c45155a: Deleting consensus metadata
I20260806 02:40:58.865954 8625 ts_tablet_manager.cc:1905] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:40:58.867971 8625 ts_tablet_manager.cc:1918] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:40:58.868090 8625 log.cc:1196] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/15371dc844d242129a09a3433bf057a7
I20260806 02:40:58.868325 8625 ts_tablet_manager.cc:1939] T 15371dc844d242129a09a3433bf057a7 P e961a77795744efa823a5df081a7375b: Deleting consensus metadata
I20260806 02:40:58.868366 7804 catalog_manager.cc:4841] TS 8d35be173cd140b1a6df29da1c45155a (127.28.191.2:34327): tablet 15371dc844d242129a09a3433bf057a7 (table test-table [id=838a55b0e194411a9c733acd8330e010]) successfully deleted
I20260806 02:40:58.868925 7809 catalog_manager.cc:4841] TS e961a77795744efa823a5df081a7375b (127.28.191.3:34623): tablet 15371dc844d242129a09a3433bf057a7 (table test-table [id=838a55b0e194411a9c733acd8330e010]) successfully deleted
W20260806 02:40:58.880010 7809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:37411 (error 108)
I20260806 02:40:58.880200 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 8132
I20260806 02:40:58.890187 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:40:58.889685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37411 (local address 127.0.0.1:50242)
0806 02:40:58.889953 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:40:58.890046 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
W20260806 02:40:58.890388 7809 catalog_manager.cc:4584] TS 64145fe1927c47528bf330fda7f444b2 (127.28.191.1:37411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15371dc844d242129a09a3433bf057a7: Network error: Client connection negotiation failed: client connection to 127.28.191.1:37411: connect: Connection refused (error 111)
I20260806 02:40:58.896507 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 8343
I20260806 02:40:58.906190 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 7785
2026-08-06T02:40:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3423 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[8735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[8735](info): set up 2 sockets
Aug 06 02:40:58 dist-test-slave-tfpf krb5kdc[8735](info): commencing operation
krb5kdc: starting...
W20260806 02:41:00.134128 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:41:00 dist-test-slave-tfpf krb5kdc[8735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:41:00.258316 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38645
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:00.368846 8751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:00.369153 8751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:00.369269 8751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:00.373167 8751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:00.373250 8751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:00.373291 8751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:00.373327 8751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:00.373359 8751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:00.378588 8751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:00.379539 8751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:00.380494 8751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:00.389168 8763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:00.389243 8751 server_base.cc:1034] running on GCE node
W20260806 02:41:00.389170 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:00.389168 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:00.389945 8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:00.390950 8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:00.392146 8751 hybrid_clock.cc:648] HybridClock initialized: now 1785984060392129 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:41:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1491 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:41:00 dist-test-slave-tfpf krb5kdc[8769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:41:00 dist-test-slave-tfpf krb5kdc[8769](info): set up 2 sockets
Aug 06 02:41:00 dist-test-slave-tfpf krb5kdc[8769](info): commencing operation
krb5kdc: starting...
W20260806 02:41:01.629930 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:41:01 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:41:01.774072 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36979
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:01.891852 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:01.892151 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:01.892194 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:01.896422 8797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:01.896500 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:01.896521 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:01.896538 8797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:01.896600 8797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:01.902182 8797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:01.903342 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:01.904443 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:01.913482 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:01.913496 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:01.913514 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:01.913964 8797 server_base.cc:1034] running on GCE node
I20260806 02:41:01.914587 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:01.915740 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:01.916944 8797 hybrid_clock.cc:648] HybridClock initialized: now 1785984061916928 us; error 36 us; skew 500 ppm
Aug 06 02:41:01 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:01.920316 8797 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:41:01.921391 8797 webserver.cc:460] Webserver started at http://127.28.191.62:35861/ using document root <none> and password file <none>
I20260806 02:41:01.922075 8797 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:01.922142 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:01.922387 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:01.924580 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "d3f9d8b121704122a08a29affbaecddd"
format_stamp: "Formatted at 2026-08-06 02:41:01 on dist-test-slave-tfpf"
server_key: "a506283f00ec96c11d4cb3d7380dd4c1"
server_key_iv: "2fb32b8f0e3ca16f0b703607066d4608"
server_key_version: "encryptionkey@0"
I20260806 02:41:01.925247 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "d3f9d8b121704122a08a29affbaecddd"
format_stamp: "Formatted at 2026-08-06 02:41:01 on dist-test-slave-tfpf"
server_key: "a506283f00ec96c11d4cb3d7380dd4c1"
server_key_iv: "2fb32b8f0e3ca16f0b703607066d4608"
server_key_version: "encryptionkey@0"
I20260806 02:41:01.929481 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:41:01.932252 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:01.933784 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:01.933954 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "d3f9d8b121704122a08a29affbaecddd"
format_stamp: "Formatted at 2026-08-06 02:41:01 on dist-test-slave-tfpf"
server_key: "a506283f00ec96c11d4cb3d7380dd4c1"
server_key_iv: "2fb32b8f0e3ca16f0b703607066d4608"
server_key_version: "encryptionkey@0"
I20260806 02:41:01.934104 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:01.947530 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:01.950994 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:01.951257 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:01.959406 8797 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36979
I20260806 02:41:01.959421 8890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36979 every 8 connection(s)
I20260806 02:41:01.960502 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:41:01.962628 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 8797
I20260806 02:41:01.962798 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:41:01.963141 29436 external_mini_cluster.cc:1428] Setting key 8f2c02152ac6bceb376699fd1227feeb
I20260806 02:41:01.963892 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:01.971239 8892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Bootstrap starting.
Aug 06 02:41:01 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:01.974254 8892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:01.975198 8892 log.cc:824] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:01.977802 8892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: No bootstrap required, opened a new log
I20260806 02:41:01.979236 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:01.964746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.28.191.62:36979)
0806 02:41:01.965321 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:01.965337 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:01.965376 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:41:01.966135 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:01.966154 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:01.966511 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:01.966881 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:01.967765 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:01.968845 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:01.969906 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:01.970394 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:01.970587 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:01.973583 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:01.973589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:01.973604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:01.973634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:01.975870 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:01.976542 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:01.976547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:01.976552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:01.976681 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:01.977020 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:01.977023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:01.977024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:01.977625 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:01.977790 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:01.978122 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 02:41:01.978475 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":204,"threads_started":1}
I20260806 02:41:01.982081 8892 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:01.982338 8892 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:01.982424 8892 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f9d8b121704122a08a29affbaecddd, State: Initialized, Role: FOLLOWER
I20260806 02:41:01.982897 8892 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:01.983071 8892 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:41:01.983163 8892 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:41:01.983289 8892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:01.984416 8892 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:01.984736 8892 leader_election.cc:302] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f9d8b121704122a08a29affbaecddd; no voters:
I20260806 02:41:01.985092 8892 leader_election.cc:288] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:41:01.985195 8899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:41:01.985407 8899 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 LEADER]: Becoming Leader. State: Replica: d3f9d8b121704122a08a29affbaecddd, State: Running, Role: LEADER
I20260806 02:41:01.985772 8899 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:01.986532 8892 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:41:01.987814 8900 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3f9d8b121704122a08a29affbaecddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } } }
I20260806 02:41:01.987964 8900 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:01.988265 8901 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f9d8b121704122a08a29affbaecddd. Latest consensus state: current_term: 1 leader_uuid: "d3f9d8b121704122a08a29affbaecddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } } }
I20260806 02:41:01.988353 8901 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: This master's current role is: LEADER
W20260806 02:41:01.992017 8917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:41:01.992132 8917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:41:01.992213 8910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:41:01.992937 8910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:41:01.998900 8910 catalog_manager.cc:1347] Generated new cluster ID: 0bd9274dde9d4b30b3fe6a5688cad48b
I20260806 02:41:01.998999 8910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:41:02.006848 8910 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:41:02.008260 8910 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:41:02.016705 8910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Generated new TSK 0
I20260806 02:41:02.017694 8910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:41:02.054955 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:02.177158 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:02.177474 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:02.177538 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:02.181804 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:02.181888 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:02.181974 8922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:41:02.187795 8922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:02.189208 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:02.190357 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:02.201077 8929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:02.201198 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:02.201128 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:02.201623 8922 server_base.cc:1034] running on GCE node
I20260806 02:41:02.202157 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:02.203024 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:02.204236 8922 hybrid_clock.cc:648] HybridClock initialized: now 1785984062204212 us; error 34 us; skew 500 ppm
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:02.207463 8922 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:41:02.208439 8922 webserver.cc:460] Webserver started at http://127.28.191.1:45301/ using document root <none> and password file <none>
I20260806 02:41:02.209239 8922 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:02.209357 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:02.209681 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:02.211947 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "37600400a4d14cb6bb8f2014bbacfbfd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "9cf68cf1cf635f67b7966468bfc7e965"
server_key_iv: "fed6f3790bfa64b5a3ab8a3983eb4c6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.212590 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "37600400a4d14cb6bb8f2014bbacfbfd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "9cf68cf1cf635f67b7966468bfc7e965"
server_key_iv: "fed6f3790bfa64b5a3ab8a3983eb4c6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.216887 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 02:41:02.219869 8940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.221465 8922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:02.221640 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "37600400a4d14cb6bb8f2014bbacfbfd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "9cf68cf1cf635f67b7966468bfc7e965"
server_key_iv: "fed6f3790bfa64b5a3ab8a3983eb4c6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.221791 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:02.240715 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:02.244514 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:02.244771 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:02.245548 8922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:02.246490 8922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:02.246565 8922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.246650 8922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:02.246702 8922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.256124 8922 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:36813
I20260806 02:41:02.256147 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:36813 every 8 connection(s)
I20260806 02:41:02.257222 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:41:02.264057 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 8922
I20260806 02:41:02.264216 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:41:02.264564 29436 external_mini_cluster.cc:1428] Setting key b6dca6dbe549754d9dbc4e4295edc34f
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:02.274966 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.259405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:34283 (local address 127.28.191.62:36979)
0806 02:41:02.259600 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:02.259605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:02.260513 (+ 908us) server_negotiation.cc:407] Connection header received
0806 02:41:02.262218 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:02.262222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:02.262277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:02.262361 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:02.263809 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.264677 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.266721 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.267130 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.267266 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.270661 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:02.270665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:02.270667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:02.270696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:02.272525 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.273190 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.273200 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.273202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.273289 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.273693 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.273696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.273698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.273913 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:02.274028 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:02.274601 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 02:41:02.274779 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:41:02.275914 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.259756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:34283)
0806 02:41:02.260318 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:02.260372 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:02.261935 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:02.262564 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:02.262572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:02.263031 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:02.263603 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.263620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.264890 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.264894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.266580 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.266593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.267339 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.267342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.267531 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.268315 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:02.268334 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:02.270425 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:02.272683 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.272690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.272718 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.273032 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.273400 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.273403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.273405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.273586 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.274028 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:02.274039 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:02.274333 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 02:41:02.274544 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 02:41:02.274938 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:41:02.277436 9116 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36979
I20260806 02:41:02.277742 9116 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:02.278280 9116 heartbeater.cc:511] Master 127.28.191.62:36979 requested a full tablet report, sending...
I20260806 02:41:02.279968 8842 ts_manager.cc:194] Registered new tserver with Master: 37600400a4d14cb6bb8f2014bbacfbfd (127.28.191.1:36813)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:41:02.281564 8842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:34283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:41:02.301034 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:02.437186 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:02.437517 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:02.437568 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:02.442111 9121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:02.442189 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:02.442255 9121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:41:02.447966 9121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:02.449266 9121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:02.450394 9121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:02.461192 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:02.461231 9131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:02.461216 9134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:02.461483 9121 server_base.cc:1034] running on GCE node
I20260806 02:41:02.461995 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:02.462720 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:02.463929 9121 hybrid_clock.cc:648] HybridClock initialized: now 1785984062463880 us; error 74 us; skew 500 ppm
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:02.467350 9121 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:41:02.468336 9121 webserver.cc:460] Webserver started at http://127.28.191.2:35213/ using document root <none> and password file <none>
I20260806 02:41:02.468914 9121 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:02.468977 9121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:02.469229 9121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:02.471568 9121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "b1f45c97e9934c328aea7cdb1fd4337e"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "1d0a40db8127b256e3befcf0cde06d38"
server_key_iv: "ac89a0fdc369827d2c648947161bb51b"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.472162 9121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "b1f45c97e9934c328aea7cdb1fd4337e"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "1d0a40db8127b256e3befcf0cde06d38"
server_key_iv: "ac89a0fdc369827d2c648947161bb51b"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.476477 9121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:41:02.479491 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.481222 9121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:02.481369 9121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "b1f45c97e9934c328aea7cdb1fd4337e"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "1d0a40db8127b256e3befcf0cde06d38"
server_key_iv: "ac89a0fdc369827d2c648947161bb51b"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.481499 9121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:02.496831 9121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:02.500629 9121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:02.500857 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:02.501614 9121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:02.502550 9121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:02.502602 9121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.502645 9121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:02.502661 9121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.512633 9121 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:37147
I20260806 02:41:02.512727 9318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:37147 every 8 connection(s)
I20260806 02:41:02.513718 9121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:41:02.521028 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 9121
I20260806 02:41:02.521230 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:41:02.521533 29436 external_mini_cluster.cc:1428] Setting key 37206af1ab0d987cc994d6dae7ca4712
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:02.529883 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.515870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:34835 (local address 127.28.191.62:36979)
0806 02:41:02.516058 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:02.516063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:02.517112 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 02:41:02.518592 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:02.518596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:02.518660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:02.518771 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:02.520115 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.520668 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.522168 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.522445 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.522534 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.525548 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:02.525552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:02.525555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:02.525593 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:02.527433 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.528028 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.528032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.528034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.528135 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.528583 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.528587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.528588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.528804 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:02.528918 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:02.529565 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 02:41:02.529712 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:41:02.530804 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.516311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:34835)
0806 02:41:02.516878 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:02.516943 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:02.518329 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:02.518947 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:02.518954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:02.519353 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:02.519940 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.519955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.520818 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.520820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.521988 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.522003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.522659 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.522665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.522882 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.523660 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:02.523678 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:02.525319 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:02.527573 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.527587 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.527607 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.527902 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.528234 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.528238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.528240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.528462 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.528918 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:02.528927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:02.529300 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 02:41:02.529506 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:41:02.529877 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260806 02:41:02.532141 9320 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36979
I20260806 02:41:02.532424 9320 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:41:02.533012 9320 heartbeater.cc:511] Master 127.28.191.62:36979 requested a full tablet report, sending...
I20260806 02:41:02.534194 8842 ts_manager.cc:194] Registered new tserver with Master: b1f45c97e9934c328aea7cdb1fd4337e (127.28.191.2:37147)
I20260806 02:41:02.534798 8842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:34835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:41:02.557723 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:02.683231 9327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:02.683547 9327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:02.683593 9327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:02.688346 9327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:02.688427 9327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:02.688519 9327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:41:02.694317 9327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:02.695574 9327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:02.696705 9327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:02.707926 9339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:02.707932 9341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:02.708020 9327 server_base.cc:1034] running on GCE node
W20260806 02:41:02.707932 9338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:02.708616 9327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:02.709398 9327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:02.710593 9327 hybrid_clock.cc:648] HybridClock initialized: now 1785984062710555 us; error 64 us; skew 500 ppm
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:02.713809 9327 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:41:02.714866 9327 webserver.cc:460] Webserver started at http://127.28.191.3:46213/ using document root <none> and password file <none>
I20260806 02:41:02.715525 9327 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:02.715631 9327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:02.715911 9327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:02.718268 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "2565e38472784fb7a0f008ac21b8c3dd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "e6860e189f163e0d07d827d524185d7a"
server_key_iv: "324d56f96e5b8440f38906b41627d138"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.718916 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "2565e38472784fb7a0f008ac21b8c3dd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "e6860e189f163e0d07d827d524185d7a"
server_key_iv: "324d56f96e5b8440f38906b41627d138"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.723357 9327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:41:02.726366 9354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.727833 9327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:02.728003 9327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "2565e38472784fb7a0f008ac21b8c3dd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "e6860e189f163e0d07d827d524185d7a"
server_key_iv: "324d56f96e5b8440f38906b41627d138"
server_key_version: "encryptionkey@0"
I20260806 02:41:02.728147 9327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:02.738590 9327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:02.742295 9327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:02.742533 9327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:02.743193 9327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:02.744117 9327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:02.744190 9327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.744273 9327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:02.744331 9327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:02.754347 9327 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36559
I20260806 02:41:02.754366 9525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36559 every 8 connection(s)
I20260806 02:41:02.755388 9327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:41:02.756731 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 9327
I20260806 02:41:02.756887 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:41:02.757283 29436 external_mini_cluster.cc:1428] Setting key ccac2432b53c14272df20dff0e327750
Aug 06 02:41:02 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984062, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:02.771478 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.757845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:44149 (local address 127.28.191.62:36979)
0806 02:41:02.758010 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:02.758015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:02.759323 (+ 1308us) server_negotiation.cc:407] Connection header received
0806 02:41:02.761197 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:02.761201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:02.761260 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:02.761340 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:02.762690 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.763246 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.764487 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.764791 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.764894 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.767687 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:02.767691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:02.767694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:02.767730 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:02.769141 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.769716 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.769720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.769722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.769815 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:02.770179 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:02.770184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:02.770186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:02.770395 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:02.770506 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:02.771096 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 02:41:02.771310 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:41:02.772663 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:02.758314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:44149)
0806 02:41:02.759151 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:02.759220 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:02.760916 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:02.761528 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:02.761535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:02.761984 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:02.762523 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.762538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.763391 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.763395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.764338 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:02.764349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:02.764947 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:02.764951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:02.765210 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:02.765897 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:02.765916 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:02.767475 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:02.769277 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.769284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.769303 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.769598 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.769910 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:02.769914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:02.769916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:02.770074 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:02.770505 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:02.770514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:02.770803 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 02:41:02.771018 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 02:41:02.771445 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":147,"threads_started":1}
I20260806 02:41:02.773838 9528 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36979
I20260806 02:41:02.774080 9528 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:02.774578 9528 heartbeater.cc:511] Master 127.28.191.62:36979 requested a full tablet report, sending...
I20260806 02:41:02.775657 8842 ts_manager.cc:194] Registered new tserver with Master: 2565e38472784fb7a0f008ac21b8c3dd (127.28.191.3:36559)
I20260806 02:41:02.776379 8842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:44149
I20260806 02:41:02.785681 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
Entry for principal oryx/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab.
WARNING: no policy specified for oryx/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
Entry for principal oryx/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab.
WARNING: no policy specified for oryx/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
Entry for principal oryx/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab.
I20260806 02:41:02.880061 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 8922
I20260806 02:41:02.903512 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 9121
I20260806 02:41:02.916473 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 9327
I20260806 02:41:02.931129 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 8797
I20260806 02:41:02.949316 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36979
--webserver_interface=127.28.191.62
--webserver_port=35861
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:03.076232 9542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:03.076570 9542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:03.076645 9542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:03.080942 9542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:03.081032 9542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:03.081051 9542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:03.081139 9542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:03.081174 9542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:03.086975 9542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=35861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:03.088182 9542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:03.089329 9542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:03.098731 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.098894 9542 server_base.cc:1034] running on GCE node
W20260806 02:41:03.098801 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.098731 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.099761 9542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:03.100921 9542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:03.102099 9542 hybrid_clock.cc:648] HybridClock initialized: now 1785984063102074 us; error 37 us; skew 500 ppm
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984063, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:03.106746 9542 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:41:03.107803 9542 webserver.cc:460] Webserver started at http://127.28.191.62:35861/ using document root <none> and password file <none>
I20260806 02:41:03.108446 9542 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:03.108563 9542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:03.112809 9542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:03.115232 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.116511 9542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:41:03.116685 9542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "d3f9d8b121704122a08a29affbaecddd"
format_stamp: "Formatted at 2026-08-06 02:41:01 on dist-test-slave-tfpf"
server_key: "a506283f00ec96c11d4cb3d7380dd4c1"
server_key_iv: "2fb32b8f0e3ca16f0b703607066d4608"
server_key_version: "encryptionkey@0"
I20260806 02:41:03.117352 9542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:03.137841 9542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:03.141383 9542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:03.141638 9542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:03.149485 9542 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36979
I20260806 02:41:03.149513 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36979 every 8 connection(s)
I20260806 02:41:03.150664 9542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:41:03.155330 9651 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:41:03.157259 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Bootstrap starting.
I20260806 02:41:03.157713 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 9542
I20260806 02:41:03.158279 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:36813
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:36979
--webserver_port=45301
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:41:03.176220 9651 log.cc:824] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:03.183609 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:41:03.184054 9651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Bootstrap complete.
I20260806 02:41:03.188908 9651 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:03.189428 9651 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f9d8b121704122a08a29affbaecddd, State: Initialized, Role: FOLLOWER
I20260806 02:41:03.190040 9651 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:03.190176 9651 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:41:03.190269 9651 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:41:03.190395 9651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:41:03.192531 9651 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:03.192932 9651 leader_election.cc:302] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f9d8b121704122a08a29affbaecddd; no voters:
I20260806 02:41:03.193397 9651 leader_election.cc:288] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:41:03.193502 9656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:41:03.193717 9656 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [term 2 LEADER]: Becoming Leader. State: Replica: d3f9d8b121704122a08a29affbaecddd, State: Running, Role: LEADER
I20260806 02:41:03.194165 9656 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } }
I20260806 02:41:03.194839 9651 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:41:03.195261 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f9d8b121704122a08a29affbaecddd. Latest consensus state: current_term: 2 leader_uuid: "d3f9d8b121704122a08a29affbaecddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } } }
I20260806 02:41:03.195385 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:03.195806 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d3f9d8b121704122a08a29affbaecddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f9d8b121704122a08a29affbaecddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36979 } } }
I20260806 02:41:03.196005 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:03.195812 9661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:41:03.200273 9661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:41:03.201136 9661 catalog_manager.cc:1259] Loaded cluster ID: 0bd9274dde9d4b30b3fe6a5688cad48b
I20260806 02:41:03.201270 9661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:41:03.203792 9661 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:41:03.204530 9661 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3f9d8b121704122a08a29affbaecddd: Loaded TSK: 0
I20260806 02:41:03.205814 9661 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:41:03.287029 9653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:03.287309 9653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:03.287356 9653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:03.291744 9653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:03.291829 9653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:03.291894 9653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:41:03.298276 9653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.1.keytab
--principal=oryx/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=45301
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:03.299510 9653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:03.300575 9653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:03.311081 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.311081 9684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.311385 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.311529 9653 server_base.cc:1034] running on GCE node
I20260806 02:41:03.312047 9653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:03.312762 9653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:03.313988 9653 hybrid_clock.cc:648] HybridClock initialized: now 1785984063313973 us; error 37 us; skew 500 ppm
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:03.317233 9653 init.cc:377] Logged in from keytab as oryx/127.28.191.1@KRBTEST.COM (short username oryx)
I20260806 02:41:03.318490 9653 webserver.cc:460] Webserver started at http://127.28.191.1:45301/ using document root <none> and password file <none>
I20260806 02:41:03.319175 9653 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:03.319265 9653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:03.323890 9653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:41:03.326277 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.327705 9653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:03.327863 9653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "37600400a4d14cb6bb8f2014bbacfbfd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "9cf68cf1cf635f67b7966468bfc7e965"
server_key_iv: "fed6f3790bfa64b5a3ab8a3983eb4c6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:03.328464 9653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:03.348145 9653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:03.351984 9653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:03.352232 9653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:03.352905 9653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:03.353909 9653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:03.353988 9653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.354059 9653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:03.354105 9653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.364162 9653 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:36813
I20260806 02:41:03.364200 9867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:36813 every 8 connection(s)
I20260806 02:41:03.365317 9653 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:41:03.367277 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 9653
I20260806 02:41:03.367838 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:37147
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:36979
--webserver_port=35213
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:41:03.382238 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.367688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:51817 (local address 127.28.191.62:36979)
0806 02:41:03.368654 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.368678 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.368728 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:41:03.370191 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.370209 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.370665 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.371086 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.372725 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.373955 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.375998 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.376532 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.376696 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.381389 (+ 4693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:51817: BlockingRecv error: recv got EOF from 127.28.191.1:51817 (error 108)
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":134,"threads_started":1}
I20260806 02:41:03.382251 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.367765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:51817)
0806 02:41:03.368252 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.368309 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.369688 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.371174 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.371181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.371721 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.372451 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.372486 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.374141 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.374148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.375818 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.375838 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.376775 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.376801 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.377104 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.377861 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.377892 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.381117 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
W20260806 02:41:03.383062 9868 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:36979 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.191.62:36979: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.389627 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.383571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:50303)
0806 02:41:03.383802 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.383815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.383892 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.384156 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.384158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.384304 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.384412 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.384417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.385542 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.385546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.386105 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.386113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.386693 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.386695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.386770 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.387357 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.387369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.389512 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260806 02:41:03.390080 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.383705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:50303 (local address 127.28.191.62:36979)
0806 02:41:03.383848 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.383853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.383870 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:41:03.383986 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.383989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.384047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.384143 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.384522 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.385397 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.386220 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.386579 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.386693 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.389719 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:50303: BlockingRecv error: recv got EOF from 127.28.191.1:50303 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.396349 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.390326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:52855)
0806 02:41:03.390459 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.390474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.390555 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.390838 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.390841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.391014 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.391111 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.391117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.391825 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.391829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.392728 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.392740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.393354 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.393358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.393457 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.394038 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.394052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.396239 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:41:03.396651 9868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:41:03.396813 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.390508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:52855 (local address 127.28.191.62:36979)
0806 02:41:03.390629 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.390632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.390645 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:41:03.390687 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.390689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.390727 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.390797 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.391223 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.391717 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.392876 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.393212 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.393321 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.396418 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:52855: BlockingRecv error: recv got EOF from 127.28.191.1:52855 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:41:03.495496 9875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:03.495918 9875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:03.495968 9875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:03.500283 9875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:03.500367 9875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:03.500432 9875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:41:03.506155 9875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.2.keytab
--principal=oryx/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=35213
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:03.507413 9875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:03.508627 9875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:03.519088 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.519047 9881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.519026 9875 server_base.cc:1034] running on GCE node
W20260806 02:41:03.519032 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.519712 9875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:03.520404 9875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:03.521575 9875 hybrid_clock.cc:648] HybridClock initialized: now 1785984063521555 us; error 30 us; skew 500 ppm
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:03.524595 9875 init.cc:377] Logged in from keytab as oryx/127.28.191.2@KRBTEST.COM (short username oryx)
I20260806 02:41:03.525592 9875 webserver.cc:460] Webserver started at http://127.28.191.2:35213/ using document root <none> and password file <none>
I20260806 02:41:03.526157 9875 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:03.526237 9875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:03.530483 9875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:41:03.532694 9896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.534041 9875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:41:03.534157 9875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "b1f45c97e9934c328aea7cdb1fd4337e"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "1d0a40db8127b256e3befcf0cde06d38"
server_key_iv: "ac89a0fdc369827d2c648947161bb51b"
server_key_version: "encryptionkey@0"
I20260806 02:41:03.534714 9875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:03.564383 9875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:03.567783 9875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:03.567955 9875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:03.568529 9875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:03.569438 9875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:03.569490 9875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.569532 9875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:03.569551 9875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.579356 9875 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:37147
I20260806 02:41:03.579383 10055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:37147 every 8 connection(s)
I20260806 02:41:03.580536 9875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:41:03.588006 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 9875
I20260806 02:41:03.588487 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:36559
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:36979
--webserver_port=46213
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:41:03.592689 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.582532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:36835 (local address 127.28.191.62:36979)
0806 02:41:03.582715 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.582719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.583592 (+ 873us) server_negotiation.cc:407] Connection header received
0806 02:41:03.584956 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.584960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.585025 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.585166 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.586367 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.586902 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.588087 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.588366 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.588476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.592560 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:36835: BlockingRecv error: recv got EOF from 127.28.191.2:36835 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:41:03.593472 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.582946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:36835)
0806 02:41:03.583420 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.583462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.584730 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.585355 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.585362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.585724 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.586208 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.586222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.587053 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.587056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.587925 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.587935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.588533 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.588537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.588771 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.589454 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.589471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.592433 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":111,"threads_started":1}
W20260806 02:41:03.594159 10056 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:36979 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.191.62:36979: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.601505 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.594747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:34347)
0806 02:41:03.594978 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.594994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.595121 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.595404 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.595410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.595618 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.595773 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.595784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.596626 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.596633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.597785 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.597802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.598309 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.598316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.598440 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.598982 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.598999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.601317 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260806 02:41:03.601629 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.594830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:34347 (local address 127.28.191.62:36979)
0806 02:41:03.594963 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.594968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.595044 (+ 76us) server_negotiation.cc:407] Connection header received
0806 02:41:03.595243 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.595246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.595287 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.595349 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.595951 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.596502 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.597939 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.598169 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.598268 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.601465 (+ 3197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:34347: BlockingRecv error: recv got EOF from 127.28.191.2:34347 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.607965 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.602141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:48785)
0806 02:41:03.602339 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.602357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.602502 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.602827 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.602833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.603023 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.603168 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.603180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.603894 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.603898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.604767 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.604777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.605324 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.605327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.605413 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.605901 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.605915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.607815 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260806 02:41:03.608073 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.602247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:48785 (local address 127.28.191.62:36979)
0806 02:41:03.602368 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.602371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.602410 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:41:03.602653 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.602656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.602699 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.602774 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.603288 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.603791 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.604901 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.605211 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.605302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.607931 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:48785: BlockingRecv error: recv got EOF from 127.28.191.2:48785 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:41:03.608289 10056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:41:03.712499 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:03.712803 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:03.712849 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:03.717213 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:03.717280 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:03.717370 10060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:41:03.723074 10060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/krb5kdc/oryx_127.28.191.3.keytab
--principal=oryx/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:36559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=46213
--tserver_master_addrs=127.28.191.62:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:03.724332 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:03.725509 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:41:03.736145 10060 server_base.cc:1034] running on GCE node
W20260806 02:41:03.736081 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.736116 10069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:03.736081 10067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:03.736819 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:03.737597 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:03.738780 10060 hybrid_clock.cc:648] HybridClock initialized: now 1785984063738764 us; error 53 us; skew 500 ppm
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:03.742028 10060 init.cc:377] Logged in from keytab as oryx/127.28.191.3@KRBTEST.COM (short username oryx)
I20260806 02:41:03.743055 10060 webserver.cc:460] Webserver started at http://127.28.191.3:46213/ using document root <none> and password file <none>
I20260806 02:41:03.743682 10060 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:03.743804 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:03.748081 10060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:41:03.750478 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.751852 10060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:41:03.752007 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "2565e38472784fb7a0f008ac21b8c3dd"
format_stamp: "Formatted at 2026-08-06 02:41:02 on dist-test-slave-tfpf"
server_key: "e6860e189f163e0d07d827d524185d7a"
server_key_iv: "324d56f96e5b8440f38906b41627d138"
server_key_version: "encryptionkey@0"
I20260806 02:41:03.752606 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:03.770951 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:03.774540 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:03.774782 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:03.775447 10060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:03.776378 10060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:03.776454 10060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.776528 10060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:03.776572 10060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:03.787267 10060 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36559
I20260806 02:41:03.787299 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36559 every 8 connection(s)
I20260806 02:41:03.788456 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:41:03.798601 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 10060
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:41:03.801813 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.790541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:36899 (local address 127.28.191.62:36979)
0806 02:41:03.790705 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.790717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.791665 (+ 948us) server_negotiation.cc:407] Connection header received
0806 02:41:03.793235 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.793239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.793290 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.793361 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.794895 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.795661 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.797191 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.797423 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.797524 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.801676 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:36899: BlockingRecv error: recv got EOF from 127.28.191.3:36899 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:41:03.802522 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.790920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:36899)
0806 02:41:03.791465 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.791520 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.792876 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.793584 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.793592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.794024 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.794716 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.794734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.795834 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.795837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.796970 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.796981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.797609 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.797612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.797772 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.798561 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.798578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.801551 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
W20260806 02:41:03.803182 10265 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:36979 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.191.62:36979: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.809551 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.803704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:45367)
0806 02:41:03.803823 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.803838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.803920 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.804161 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.804164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.804358 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.804522 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.804529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.805392 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.805399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.806220 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.806231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.806726 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.806730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.806833 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.807392 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.807413 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.809384 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:41:03.809651 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.803773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:45367 (local address 127.28.191.62:36979)
0806 02:41:03.803883 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.803887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.803899 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:41:03.803986 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.803991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.804046 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.804119 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.804641 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.805265 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.806359 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.806595 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.806722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.809520 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:45367: BlockingRecv error: recv got EOF from 127.28.191.3:45367 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:41:03.810721 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.799828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.28.191.62:36979)
0806 02:41:03.800099 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.800125 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.800143 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:41:03.800254 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.800258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.800315 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.800417 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.801807 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.802433 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.803615 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.803963 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.804078 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.806025 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:03.806032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:03.806051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:03.806080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:03.808020 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:03.808593 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:03.808599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:03.808605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:03.808744 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:03.809089 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:03.809093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:03.809094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:03.809496 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:03.810159 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:03.810320 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 02:41:03.810602 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:03 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:03.815568 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:03.810333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:43721)
0806 02:41:03.810432 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:03.810448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:03.810526 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:03.810786 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:03.810788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:03.810945 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:03.811020 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.811025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.811825 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.811828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.812349 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:03.812357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.812884 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.812888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:03.812972 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.813525 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:03.813536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:03.815466 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 02:41:03.815706 9874 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:03.810423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:43721 (local address 127.28.191.62:36979)
0806 02:41:03.810584 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:03.810589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:03.810603 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:41:03.810642 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:03.810644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:03.810679 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:03.810769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:03.811144 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.811729 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.812462 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:03.812784 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:03.812868 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:03.815594 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:43721: BlockingRecv error: recv got EOF from 127.28.191.3:43721 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:41:03.815960 10265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:04.403859 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:04.397457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:58541)
0806 02:41:04.397812 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:04.397831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:04.397972 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:04.398353 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:04.398355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:04.398610 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:04.398760 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.398770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.399766 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.399769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.400445 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.400456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.400939 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.400942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.401025 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.401659 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:04.401672 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:04.403726 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":82,"threads_started":1}
W20260806 02:41:04.404126 10274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:04.397523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:58541 (local address 127.28.191.62:36979)
0806 02:41:04.397821 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:04.397824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:04.397950 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:41:04.398090 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:04.398093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:04.398176 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:04.398305 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:04.398896 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.399597 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.400578 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.400809 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.400894 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.403883 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:58541: BlockingRecv error: recv got EOF from 127.28.191.1:58541 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:04.614954 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:04.609151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:38443)
0806 02:41:04.609475 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:04.609489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:04.609598 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:04.609841 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:04.609843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:04.610040 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:04.610173 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.610178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.610954 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.610958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.611589 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.611600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.612104 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.612107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.612177 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.612742 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:04.612754 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:04.614819 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
W20260806 02:41:04.615123 10274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:04.609201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:38443 (local address 127.28.191.62:36979)
0806 02:41:04.609375 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:04.609379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:04.609602 (+ 223us) server_negotiation.cc:407] Connection header received
0806 02:41:04.609673 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:04.609675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:04.609724 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:04.609798 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:04.610281 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.610827 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.611730 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.611955 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.612043 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.614990 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:38443: BlockingRecv error: recv got EOF from 127.28.191.2:38443 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:04 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:04.822934 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:04.816771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:42327)
0806 02:41:04.817115 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:04.817159 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:04.817273 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:04.817544 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:04.817548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:04.817776 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:04.817872 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.817878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.818658 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.818662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.819241 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:04.819250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.819744 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.819747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:04.819832 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.820370 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:04.820383 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:04.822746 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
W20260806 02:41:04.823045 10274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:04.816854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42327 (local address 127.28.191.62:36979)
0806 02:41:04.817056 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:04.817102 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:04.817243 (+ 141us) server_negotiation.cc:407] Connection header received
0806 02:41:04.817348 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:04.817351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:04.817409 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:04.817511 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:04.818004 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.818537 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.819379 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:04.819617 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:04.819725 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:04.822917 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:42327: BlockingRecv error: recv got EOF from 127.28.191.3:42327 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:05.414428 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:05.404722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:33315)
0806 02:41:05.404994 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:05.405010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:05.405202 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:05.405530 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:05.405533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:05.405765 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:05.405870 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.405877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.406986 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.406989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.407546 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.407562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.408012 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.408015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.408080 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.411938 (+ 3858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:05.411954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:05.414243 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
W20260806 02:41:05.414582 10278 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:05.404816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:33315 (local address 127.28.191.62:36979)
0806 02:41:05.405145 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:05.405150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:05.405174 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:41:05.405293 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:05.405296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:05.405353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:05.405473 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:05.405991 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.406869 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.407663 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.407911 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.407997 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.414433 (+ 6436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:33315: BlockingRecv error: recv got EOF from 127.28.191.1:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:05.621904 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:05.615998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:49965)
0806 02:41:05.616269 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:05.616285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:05.616398 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:05.616685 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:05.616688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:05.616906 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:05.617012 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.617017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.617892 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.617895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.618506 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.618517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.619003 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.619006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.619079 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.619612 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:05.619624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:05.621744 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
W20260806 02:41:05.622107 10278 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:05.616055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:49965 (local address 127.28.191.62:36979)
0806 02:41:05.616212 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:05.616216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:05.616376 (+ 160us) server_negotiation.cc:407] Connection header received
0806 02:41:05.616485 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:05.616487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:05.616534 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:05.616600 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:05.617212 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.617755 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.618647 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.618869 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.618953 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.621930 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:49965: BlockingRecv error: recv got EOF from 127.28.191.2:49965 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:05 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:05.829955 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:05.823962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:60561)
0806 02:41:05.824275 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:05.824299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:05.824404 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:05.824688 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:05.824690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:05.824891 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:05.824992 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.824998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.825834 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.825838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.826515 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:05.826544 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.827031 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.827035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:05.827113 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.827747 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:05.827759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:05.829809 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":140,"threads_started":1}
W20260806 02:41:05.830168 10278 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:05.824185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60561 (local address 127.28.191.62:36979)
0806 02:41:05.824327 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:05.824331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:05.824353 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:41:05.824501 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:05.824504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:05.824555 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:05.824643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:05.825152 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.825703 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.826675 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:05.826906 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:05.827024 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:05.829980 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:60561: BlockingRecv error: recv got EOF from 127.28.191.3:60561 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:06.421264 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:06.415406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:40593)
0806 02:41:06.415698 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:06.415712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:06.415805 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:06.416046 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:06.416049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:06.416241 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:06.416345 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.416350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.417227 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.417230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.417804 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.417813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.418297 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.418300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.418369 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.418920 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:06.418934 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:06.421102 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
W20260806 02:41:06.421425 10285 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:06.415564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:40593 (local address 127.28.191.62:36979)
0806 02:41:06.415797 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:06.415800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:06.415815 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:41:06.415873 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:06.415875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:06.415916 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:06.416000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:06.416461 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.417042 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.417923 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.418174 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.418259 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.421285 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:40593: BlockingRecv error: recv got EOF from 127.28.191.1:40593 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:06.628944 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:06.622866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:47131)
0806 02:41:06.623196 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:06.623210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:06.623359 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:06.623654 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:06.623657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:06.623879 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:06.623975 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.623981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.624769 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.624774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.625401 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.625411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.625885 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.625889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.625973 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.626650 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:06.626663 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:06.628811 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
W20260806 02:41:06.629189 10285 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:06.622929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:47131 (local address 127.28.191.62:36979)
0806 02:41:06.623098 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:06.623103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:06.623295 (+ 192us) server_negotiation.cc:407] Connection header received
0806 02:41:06.623468 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:06.623472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:06.623526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:06.623625 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:06.624105 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.624651 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.625531 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.625763 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.625879 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.629012 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:47131: BlockingRecv error: recv got EOF from 127.28.191.2:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:06 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:06.837285 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:06.830938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:57507)
0806 02:41:06.831295 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:06.831309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:06.831456 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:06.831823 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:06.831826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:06.832090 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:06.832205 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.832212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.833095 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.833106 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.833729 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:06.833739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.834221 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.834224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:06.834296 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.834840 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:06.834859 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:06.837138 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
W20260806 02:41:06.837500 10285 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:06.831026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:57507 (local address 127.28.191.62:36979)
0806 02:41:06.831181 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:06.831187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:06.831392 (+ 205us) server_negotiation.cc:407] Connection header received
0806 02:41:06.831600 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:06.831603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:06.831695 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:06.831795 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:06.832327 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.832904 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.833851 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:06.834113 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:06.834215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:06.837348 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:57507: BlockingRecv error: recv got EOF from 127.28.191.3:57507 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:07.428402 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:07.422213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:49317)
0806 02:41:07.422553 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:07.422568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:07.422669 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:07.423002 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:07.423005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:07.423227 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:07.423334 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.423340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.424186 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.424189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.424799 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.424807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.425422 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.425426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.425501 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.426161 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:07.426174 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:07.428266 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
W20260806 02:41:07.428755 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:07.422315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:49317 (local address 127.28.191.62:36979)
0806 02:41:07.422604 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:07.422608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:07.422624 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:41:07.422769 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:07.422772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:07.422833 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:07.422921 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:07.423466 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.424065 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.424931 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.425298 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.425394 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.428575 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:49317: BlockingRecv error: recv got EOF from 127.28.191.1:49317 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":71,"threads_started":1}
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:07.636525 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:07.629850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:40899)
0806 02:41:07.630259 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:07.630280 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:07.630444 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:07.630784 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:07.630787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:07.631025 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:07.631165 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.631172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.632034 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.632039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.632751 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.632762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.633422 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.633426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.633526 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.634192 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:07.634206 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:07.636329 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
W20260806 02:41:07.636647 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:07.630012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:40899 (local address 127.28.191.62:36979)
0806 02:41:07.630219 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:07.630224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:07.630370 (+ 146us) server_negotiation.cc:407] Connection header received
0806 02:41:07.630592 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:07.630596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:07.630656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:07.630754 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:07.631303 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.631904 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.632894 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.633269 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.633402 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.636506 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:40899: BlockingRecv error: recv got EOF from 127.28.191.2:40899 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:07 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:07.844559 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:07.838305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:33657)
0806 02:41:07.838688 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:07.838703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:07.838814 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:07.839134 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:07.839137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:07.839337 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:07.839447 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.839452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.840292 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.840295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.840858 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:07.840867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.841423 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.841427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:07.841497 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.842038 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:07.842050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:07.844406 (+ 2356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
W20260806 02:41:07.844820 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:07.838387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:33657 (local address 127.28.191.62:36979)
0806 02:41:07.838608 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:07.838614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:07.838782 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:41:07.838899 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:07.838902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:07.838957 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:07.839038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:07.839626 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.840186 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.840982 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:07.841304 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:07.841395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:07.844658 (+ 3263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:33657: BlockingRecv error: recv got EOF from 127.28.191.3:33657 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:08.435225 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:08.429368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:45883)
0806 02:41:08.429662 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:08.429683 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:08.429784 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:08.430117 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:08.430120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:08.430331 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:08.430433 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.430439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.431210 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.431213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.431907 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.431917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.432372 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.432374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.432437 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.433151 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:08.433164 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:08.435098 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260806 02:41:08.435472 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:08.429509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:45883 (local address 127.28.191.62:36979)
0806 02:41:08.429755 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:08.429758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:08.429772 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:41:08.429880 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:08.429883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:08.429927 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:08.430021 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:08.430548 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.431098 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.432022 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.432260 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.432352 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.435309 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:45883: BlockingRecv error: recv got EOF from 127.28.191.1:45883 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:08.644438 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:08.637658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:56325)
0806 02:41:08.638178 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:08.638196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:08.638368 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:08.638679 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:08.638681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:08.638896 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:08.639066 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.639076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.639906 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.639909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.640485 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.640494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.641360 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.641364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.641455 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.642097 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:08.642109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:08.644256 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
W20260806 02:41:08.644491 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:08.637914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:56325 (local address 127.28.191.62:36979)
0806 02:41:08.638075 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:08.638079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:08.638345 (+ 266us) server_negotiation.cc:407] Connection header received
0806 02:41:08.638468 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:08.638471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:08.638546 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:08.638631 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:08.639209 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.639776 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.640637 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.641042 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.641190 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.644384 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:56325: BlockingRecv error: recv got EOF from 127.28.191.2:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:08 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:08.852087 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:08.845666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:47379)
0806 02:41:08.845965 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:08.845980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:08.846092 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:08.846380 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:08.846382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:08.846592 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:08.846695 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.846701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.847540 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.847543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.848172 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:08.848180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.848674 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.848679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:08.848759 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.849409 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:08.849425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:08.851899 (+ 2474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260806 02:41:08.852197 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:08.845878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:47379 (local address 127.28.191.62:36979)
0806 02:41:08.846048 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:08.846052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:08.846074 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:41:08.846178 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:08.846181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:08.846238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:08.846340 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:08.846850 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.847416 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.848294 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:08.848517 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:08.848614 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:08.852076 (+ 3462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:47379: BlockingRecv error: recv got EOF from 127.28.191.3:47379 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:09.442559 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:09.436192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:34595)
0806 02:41:09.436523 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:09.436545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:09.436648 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:09.436954 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:09.436957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:09.437183 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:09.437288 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.437294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.438169 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.438173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.438948 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.438960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.439513 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.439517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.439622 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.440258 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:09.440271 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:09.442434 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
W20260806 02:41:09.443005 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:09.436309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:34595 (local address 127.28.191.62:36979)
0806 02:41:09.436639 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:09.436643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:09.436659 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:41:09.436745 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:09.436748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:09.436797 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:09.436887 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:09.437411 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.437996 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.439054 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.439385 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.439506 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.442861 (+ 3355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:34595: BlockingRecv error: recv got EOF from 127.28.191.1:34595 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":88,"threads_started":1}
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:09.652153 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:09.645646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:38541)
0806 02:41:09.645977 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:09.645992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:09.646105 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:09.646429 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:09.646431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:09.646662 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:09.646769 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.646776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.647695 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.647699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.648624 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.648639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.649210 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.649213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.649283 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.649811 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:09.649823 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:09.652000 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
W20260806 02:41:09.652302 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:09.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:38541 (local address 127.28.191.62:36979)
0806 02:41:09.645980 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:09.645984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:09.646075 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:41:09.646198 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:09.646201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:09.646253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:09.646347 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:09.646925 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.647585 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.648781 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.649046 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.649205 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.652171 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:38541: BlockingRecv error: recv got EOF from 127.28.191.2:38541 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:09 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:09.859254 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:09.853024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:40935)
0806 02:41:09.853406 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:09.853422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:09.853540 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:09.853883 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:09.853885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:09.854082 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:09.854192 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.854197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.854971 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.854974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.855542 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:09.855550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.856061 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.856064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:09.856131 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.856847 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:09.856859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:09.859132 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
W20260806 02:41:09.859476 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:09.853181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:40935 (local address 127.28.191.62:36979)
0806 02:41:09.853319 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:09.853330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:09.853498 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:41:09.853642 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:09.853644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:09.853694 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:09.853776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:09.854325 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.854861 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.855678 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:09.855933 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:09.856037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:09.859318 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:40935: BlockingRecv error: recv got EOF from 127.28.191.3:40935 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:10.455032 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:10.443456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:45879)
0806 02:41:10.443831 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:10.443845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:10.443947 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:10.444263 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:10.444265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:10.444459 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:10.444557 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.444562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.445344 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.445347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.445889 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.445896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.446335 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.446338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.446402 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.452579 (+ 6177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:10.452596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:10.454868 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":92,"threads_started":1}
W20260806 02:41:10.455295 10305 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:10.443537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:45879 (local address 127.28.191.62:36979)
0806 02:41:10.443847 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:10.443850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:10.443920 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:41:10.444084 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:10.444087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:10.444139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:10.444226 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:10.444674 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.445249 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.445994 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.446232 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.446314 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.455103 (+ 8789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:45879: BlockingRecv error: recv got EOF from 127.28.191.1:45879 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:10.659401 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:10.653286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:44303)
0806 02:41:10.653633 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:10.653647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:10.653755 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:10.654054 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:10.654056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:10.654261 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:10.654362 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.654367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.655216 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.655219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.655788 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.655797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.656375 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.656378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.656450 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.657168 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:10.657180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:10.659270 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
W20260806 02:41:10.659682 10305 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:10.653336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:44303 (local address 127.28.191.62:36979)
0806 02:41:10.653514 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:10.653519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:10.653734 (+ 215us) server_negotiation.cc:407] Connection header received
0806 02:41:10.653855 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:10.653858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:10.653917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:10.654013 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:10.654478 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.655080 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.655982 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.656238 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.656332 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.659435 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:44303: BlockingRecv error: recv got EOF from 127.28.191.2:44303 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:10 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:10.865780 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:10.860152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:47833)
0806 02:41:10.860504 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:10.860518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:10.860621 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:10.860844 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:10.860846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:10.861043 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:10.861207 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.861214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.861942 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.861945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.862505 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:10.862513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.862976 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.862980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:10.863058 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.863591 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:10.863603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:10.865663 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
W20260806 02:41:10.865953 10305 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:10.860217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:47833 (local address 127.28.191.62:36979)
0806 02:41:10.860347 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:10.860350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:10.860586 (+ 236us) server_negotiation.cc:407] Connection header received
0806 02:41:10.860688 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:10.860691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:10.860736 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:10.860806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:10.861300 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.861836 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.862625 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:10.862844 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:10.862927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:10.865819 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:47833: BlockingRecv error: recv got EOF from 127.28.191.3:47833 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:11.462235 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:11.455966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:33603)
0806 02:41:11.456294 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:11.456314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:11.456434 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:11.456775 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:11.456777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:11.456968 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:11.457128 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.457135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.458104 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.458107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.458777 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.458789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.459292 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.459296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.459390 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.460076 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:11.460088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:11.462054 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
W20260806 02:41:11.462316 10314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:11.456174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:33603 (local address 127.28.191.62:36979)
0806 02:41:11.456525 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:11.456529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:11.456549 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:41:11.456591 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:11.456593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:11.456637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:11.456738 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:11.457259 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.457938 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.458901 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.459161 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.459283 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.462185 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:33603: BlockingRecv error: recv got EOF from 127.28.191.1:33603 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":131,"threads_started":1}
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:11.667956 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:11.660298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:54795)
0806 02:41:11.660568 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:11.660581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:11.660694 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:11.660982 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:11.660987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:11.661289 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:11.661440 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.661452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.662821 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.662827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.663880 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.663896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.664489 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.664495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.664614 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.665234 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:11.665252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:11.667745 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260806 02:41:11.668205 10314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:11.660332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:54795 (local address 127.28.191.62:36979)
0806 02:41:11.660471 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:11.660477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:11.660660 (+ 183us) server_negotiation.cc:407] Connection header received
0806 02:41:11.660772 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:11.660775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:11.660820 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:11.660925 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:11.661580 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.662686 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.664010 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.664352 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.664466 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.667925 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:54795: BlockingRecv error: recv got EOF from 127.28.191.2:54795 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:11 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:11.873145 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:11.866782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:59061)
0806 02:41:11.867242 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:11.867267 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:11.867422 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:11.867714 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:11.867716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:11.867949 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:11.868044 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.868049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.868894 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.868898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.869641 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:11.869650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.870101 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.870104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:11.870180 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.870722 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:11.870734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:11.872911 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":103,"threads_started":1}
W20260806 02:41:11.873256 10314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:11.866967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:59061 (local address 127.28.191.62:36979)
0806 02:41:11.867212 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:11.867217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:11.867355 (+ 138us) server_negotiation.cc:407] Connection header received
0806 02:41:11.867532 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:11.867536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:11.867596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:11.867689 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:11.868195 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.868778 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.869762 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:11.869985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:11.870103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:11.873127 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:59061: BlockingRecv error: recv got EOF from 127.28.191.3:59061 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:12.468930 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:12.463086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:52413)
0806 02:41:12.463379 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:12.463393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:12.463486 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:12.463746 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:12.463748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:12.463935 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:12.464033 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.464038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.464813 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.464816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.465463 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.465474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.465935 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.465938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.465999 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.466523 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:12.466535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:12.468812 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
W20260806 02:41:12.469132 10318 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:12.463196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:52413 (local address 127.28.191.62:36979)
0806 02:41:12.463434 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:12.463438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:12.463451 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:41:12.463580 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:12.463583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:12.463628 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:12.463712 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:12.464156 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.464718 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.465597 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.465825 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.465909 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.468953 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:52413: BlockingRecv error: recv got EOF from 127.28.191.1:52413 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:12.674525 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:12.668870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:42787)
0806 02:41:12.669150 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:12.669165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:12.669276 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:12.669582 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:12.669585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:12.669771 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:12.669874 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.669879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.670656 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.670659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.671234 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.671242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.671695 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.671698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.671761 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.672329 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:12.672342 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:12.674387 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
W20260806 02:41:12.674736 10318 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:12.668944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:42787 (local address 127.28.191.62:36979)
0806 02:41:12.669145 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:12.669149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:12.669245 (+ 96us) server_negotiation.cc:407] Connection header received
0806 02:41:12.669360 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:12.669362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:12.669414 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:12.669493 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:12.669990 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.670518 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.671350 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.671588 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.671673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.674579 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:42787: BlockingRecv error: recv got EOF from 127.28.191.2:42787 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:12 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:12.879870 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:12.874118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:43679)
0806 02:41:12.874436 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:12.874451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:12.874566 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:12.874801 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:12.874803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:12.875023 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:12.875127 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.875133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.875933 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.875936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.876510 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:12.876520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.876974 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.876977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:12.877142 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.877685 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:12.877697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:12.879684 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
W20260806 02:41:12.879997 10318 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:12.874208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:43679 (local address 127.28.191.62:36979)
0806 02:41:12.874359 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:12.874362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:12.874529 (+ 167us) server_negotiation.cc:407] Connection header received
0806 02:41:12.874637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:12.874639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:12.874687 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:12.874756 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:12.875245 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.875776 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.876630 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:12.876863 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:12.876944 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:12.879844 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:43679: BlockingRecv error: recv got EOF from 127.28.191.3:43679 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:13.475673 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:13.469899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:45917)
0806 02:41:13.470226 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:13.470240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:13.470332 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:13.470591 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:13.470593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:13.470777 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:13.470872 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.470876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.471664 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.471667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.472406 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.472416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.473026 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.473030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.473188 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.473702 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:13.473714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:13.475534 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
W20260806 02:41:13.475817 10324 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:13.470045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:45917 (local address 127.28.191.62:36979)
0806 02:41:13.470286 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:13.470290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:13.470303 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:41:13.470431 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:13.470434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:13.470475 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:13.470563 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:13.470966 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.471542 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.472541 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.472886 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.472988 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.475684 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:45917: BlockingRecv error: recv got EOF from 127.28.191.1:45917 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":64,"threads_started":1}
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:13.681216 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:13.675416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:47893)
0806 02:41:13.675743 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:13.675756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:13.675848 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:13.676159 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:13.676161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:13.676349 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:13.676452 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.676457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.677255 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.677258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.677803 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.677810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.678309 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.678312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.678377 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.678868 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:13.678879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:13.681017 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":160,"threads_started":1}
W20260806 02:41:13.681387 10324 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:13.675507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:47893 (local address 127.28.191.62:36979)
0806 02:41:13.675702 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:13.675709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:13.675834 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:41:13.675932 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:13.675934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:13.676038 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:13.676125 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:13.676560 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.677158 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.677922 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.678188 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.678282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.681246 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:47893: BlockingRecv error: recv got EOF from 127.28.191.2:47893 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:13 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:13.886121 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:13.880699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:50507)
0806 02:41:13.881003 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:13.881017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:13.881161 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:13.881385 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:13.881387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:13.881579 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:13.881677 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.881682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.882441 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.882445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.883023 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:13.883033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.883450 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.883453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:13.883523 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.884076 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:13.884089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:13.885983 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260806 02:41:13.886262 10324 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:13.880867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:50507 (local address 127.28.191.62:36979)
0806 02:41:13.881026 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:13.881029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:13.881114 (+ 85us) server_negotiation.cc:407] Connection header received
0806 02:41:13.881228 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:13.881230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:13.881275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:13.881342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:13.881778 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.882307 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.883135 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:13.883347 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:13.883420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:13.886141 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:50507: BlockingRecv error: recv got EOF from 127.28.191.3:50507 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:14.482663 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:14.476575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:44845)
0806 02:41:14.476899 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:14.476917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:14.477078 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:14.477400 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:14.477404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:14.477640 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:14.477745 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.477751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.478591 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.478598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.479185 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.479193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.479715 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.479719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.479819 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.480551 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:14.480563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:14.482490 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
W20260806 02:41:14.482756 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:14.476647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:44845 (local address 127.28.191.62:36979)
0806 02:41:14.476956 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:14.476964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:14.476982 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:41:14.477201 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:14.477205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:14.477262 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:14.477375 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:14.477880 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.478455 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.479320 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.479582 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.479707 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.482623 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:44845: BlockingRecv error: recv got EOF from 127.28.191.1:44845 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:14.688094 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:14.682162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:57899)
0806 02:41:14.682452 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:14.682466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:14.682567 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:14.682803 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:14.682805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:14.682994 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:14.683116 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.683122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.683876 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.683878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.684557 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.684567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.685153 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.685157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.685244 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.685904 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:14.685917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:14.687918 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
W20260806 02:41:14.688159 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:14.682221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57899 (local address 127.28.191.62:36979)
0806 02:41:14.682374 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:14.682378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:14.682535 (+ 157us) server_negotiation.cc:407] Connection header received
0806 02:41:14.682643 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:14.682646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:14.682690 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:14.682762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:14.683249 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.683776 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.684670 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.684988 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.685146 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.688051 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:57899: BlockingRecv error: recv got EOF from 127.28.191.2:57899 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:14 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:14.893013 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:14.887087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:54873)
0806 02:41:14.887395 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:14.887410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:14.887536 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:14.887794 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:14.887797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:14.888009 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:14.888118 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.888125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.888843 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.888846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.889588 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:14.889599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.890045 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.890048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:14.890117 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.890714 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:14.890727 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:14.892888 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":121,"threads_started":1}
W20260806 02:41:14.893196 10333 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:14.887179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:54873 (local address 127.28.191.62:36979)
0806 02:41:14.887318 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:14.887322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:14.887487 (+ 165us) server_negotiation.cc:407] Connection header received
0806 02:41:14.887622 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:14.887624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:14.887671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:14.887743 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:14.888228 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.888739 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.889702 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:14.889918 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:14.889997 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:14.893012 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:54873: BlockingRecv error: recv got EOF from 127.28.191.3:54873 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:15.491971 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:15.483469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:42457)
0806 02:41:15.483758 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:15.483771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:15.483861 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:15.484112 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:15.484114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:15.484303 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:15.484402 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.484407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.485163 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.485165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.485720 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.485728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.486162 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.486165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.486226 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.489944 (+ 3718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:15.489957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:15.491851 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
W20260806 02:41:15.492182 10339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:15.483570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:42457 (local address 127.28.191.62:36979)
0806 02:41:15.483805 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:15.483809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:15.483821 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:41:15.483947 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:15.483950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:15.483994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:15.484075 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:15.484502 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.485042 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.485836 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.486064 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.486146 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.492001 (+ 5855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:42457: BlockingRecv error: recv got EOF from 127.28.191.1:42457 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:15.694154 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:15.688843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:42679)
0806 02:41:15.689139 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:15.689159 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:15.689261 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:15.689485 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:15.689487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:15.689665 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:15.689760 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.689765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.690480 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.690483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.691032 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.691039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.691458 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.691461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.691550 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.692027 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:15.692038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:15.694012 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
W20260806 02:41:15.694314 10339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:15.688922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:42679 (local address 127.28.191.62:36979)
0806 02:41:15.689103 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:15.689108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:15.689226 (+ 118us) server_negotiation.cc:407] Connection header received
0806 02:41:15.689340 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:15.689342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:15.689385 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:15.689450 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:15.689851 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.690388 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.691146 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.691361 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.691440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.694170 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:42679: BlockingRecv error: recv got EOF from 127.28.191.2:42679 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:15 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:15.899788 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:15.893878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:42997)
0806 02:41:15.894168 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:15.894191 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:15.894329 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:15.894614 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:15.894617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:15.894860 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:15.894956 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.894961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.895849 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.895854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.896584 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:15.896594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.897150 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.897154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:15.897236 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.897784 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:15.897795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:15.899675 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
W20260806 02:41:15.899937 10339 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:15.893931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42997 (local address 127.28.191.62:36979)
0806 02:41:15.894136 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:15.894140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:15.894265 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:41:15.894446 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:15.894449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:15.894497 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:15.894595 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:15.895112 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.895738 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.896708 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:15.896939 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:15.897158 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:15.899818 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:42997: BlockingRecv error: recv got EOF from 127.28.191.3:42997 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:16.498404 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:16.492785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:59717)
0806 02:41:16.493100 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:16.493115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:16.493216 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:16.493458 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:16.493460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:16.493657 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:16.493755 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.493760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.494494 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.494497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.495152 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.495161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.495704 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.495707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.495823 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.496386 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:16.496397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:16.498293 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
W20260806 02:41:16.498589 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:16.492836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:59717 (local address 127.28.191.62:36979)
0806 02:41:16.493098 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:16.493102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:16.493166 (+ 64us) server_negotiation.cc:407] Connection header received
0806 02:41:16.493300 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:16.493302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:16.493348 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:16.493424 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:16.493854 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.494398 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.495252 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.495597 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.495687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.498426 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:59717: BlockingRecv error: recv got EOF from 127.28.191.1:59717 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:16.701179 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:16.695097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:36477)
0806 02:41:16.695385 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:16.695408 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:16.695507 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:16.695755 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:16.695758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:16.696078 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:16.696186 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.696217 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.696969 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.696972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.697632 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.697642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.698093 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.698096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.698162 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.698844 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:16.698863 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:16.700950 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260806 02:41:16.701208 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:16.695220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:36477 (local address 127.28.191.62:36979)
0806 02:41:16.695398 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:16.695401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:16.695473 (+ 72us) server_negotiation.cc:407] Connection header received
0806 02:41:16.695593 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:16.695595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:16.695639 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:16.695709 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:16.696331 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.696869 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.697735 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.697950 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.698063 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.701108 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:36477: BlockingRecv error: recv got EOF from 127.28.191.2:36477 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:16 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:16.906162 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:16.900770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:52177)
0806 02:41:16.901097 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:16.901118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:16.901219 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:16.901456 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:16.901458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:16.901648 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:16.901746 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.901751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.902460 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.902463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.903064 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:16.903073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.903480 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.903482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:16.903551 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.904055 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:16.904067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:16.906029 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":130,"threads_started":1}
W20260806 02:41:16.906322 10347 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:16.900815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:52177 (local address 127.28.191.62:36979)
0806 02:41:16.900943 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:16.900946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:16.901186 (+ 240us) server_negotiation.cc:407] Connection header received
0806 02:41:16.901299 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:16.901302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:16.901349 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:16.901419 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:16.901845 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.902361 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.903167 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:16.903379 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:16.903454 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:16.906168 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:52177: BlockingRecv error: recv got EOF from 127.28.191.3:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:17.504772 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:17.499173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:39287)
0806 02:41:17.499504 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:17.499523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:17.499640 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:17.499895 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:17.499897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:17.500109 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:17.500214 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.500220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.500981 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.500985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.501613 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.501623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.502087 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.502090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.502163 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.502703 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:17.502715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:17.504644 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
W20260806 02:41:17.504918 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:17.499278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:39287 (local address 127.28.191.62:36979)
0806 02:41:17.499642 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:17.499645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:17.499658 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:41:17.499716 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:17.499718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:17.499760 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:17.499836 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:17.500321 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.500866 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.501741 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.501966 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.502057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.504783 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:39287: BlockingRecv error: recv got EOF from 127.28.191.1:39287 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":87,"threads_started":1}
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:17.707834 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:17.702062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:56839)
0806 02:41:17.702335 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:17.702352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:17.702486 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:17.702773 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:17.702776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:17.702965 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:17.703074 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.703080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.704163 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.704166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.704838 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.704849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.705345 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.705348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.705410 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.705925 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:17.705936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:17.707709 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
W20260806 02:41:17.708033 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:17.702105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:56839 (local address 127.28.191.62:36979)
0806 02:41:17.702254 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:17.702257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:17.702434 (+ 177us) server_negotiation.cc:407] Connection header received
0806 02:41:17.702573 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:17.702576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:17.702622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:17.702702 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:17.703215 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.704049 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.704954 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.705244 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.705338 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.707871 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:56839: BlockingRecv error: recv got EOF from 127.28.191.2:56839 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:17 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.3@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:17.913949 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:17.907037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.3:33187)
0806 02:41:17.907339 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:17.907351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:17.907485 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:17.907765 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:17.907768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:17.907981 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:17.908101 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.908106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.908873 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.908878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.909652 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:17.909662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.910132 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.910136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:17.910221 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.910855 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:17.910867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:17.913765 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
W20260806 02:41:17.913992 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:17.907065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:33187 (local address 127.28.191.62:36979)
0806 02:41:17.907194 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:17.907199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:17.907425 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:41:17.907602 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:17.907605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:17.907660 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:17.907742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:17.908221 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.908760 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.909776 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:17.910020 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:17.910129 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:17.913886 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.3:33187: BlockingRecv error: recv got EOF from 127.28.191.3:33187 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 02:41:18 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:18 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.1@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:18.511878 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:18.505904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.1:55985)
0806 02:41:18.506190 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:18.506203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:18.506290 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:18.506546 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:18.506548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:18.506736 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:18.506846 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:18.506852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.507877 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.507885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:18.508673 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:18.508685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.509182 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.509185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:18.509247 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:18.509737 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:18.509748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:18.511755 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
W20260806 02:41:18.512028 10361 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:18.505966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:55985 (local address 127.28.191.62:36979)
0806 02:41:18.506229 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:18.506233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:18.506250 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:41:18.506378 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:18.506381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:18.506425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:18.506506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:18.506948 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.507735 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.508791 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.509011 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.509176 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:18.511892 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.1:55985: BlockingRecv error: recv got EOF from 127.28.191.1:55985 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
Aug 06 02:41:18 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:41:18 dist-test-slave-tfpf krb5kdc[8769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.191.2@KRBTEST.COM for oryx/127.28.191.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:41:18.714440 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:18.708747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36979 (local address 127.28.191.2:41847)
0806 02:41:18.709031 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:18.709046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:18.709155 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:18.709387 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:18.709389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:18.709589 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:18.709684 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:18.709689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.710458 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.710461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:18.711156 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:18.711166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.711628 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.711631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:18.711692 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:18.712202 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:18.712214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:18.714254 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.191.62:36979: Server oryx/127.28.191.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
W20260806 02:41:18.714485 10361 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:18.708881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:41847 (local address 127.28.191.62:36979)
0806 02:41:18.709137 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:18.709142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:18.709168 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:41:18.709237 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:18.709239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:18.709285 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:18.709352 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:18.709796 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.710352 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.711282 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:18.711520 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:18.711621 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:18.714381 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.191.2:41847: BlockingRecv error: recv got EOF from 127.28.191.2:41847 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260806 02:41:18.805176 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 9653
I20260806 02:41:18.821995 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 9875
I20260806 02:41:18.832088 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 10060
I20260806 02:41:18.843793 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 9542
2026-08-06T02:41:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18443 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:18Z Disabled control of system clock
I20260806 02:41:18.881280 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42063
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:42063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:41:19.004335 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:19.004626 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:19.004671 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:19.008793 10368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:19.008859 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:19.008875 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:19.008893 10368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:19.008908 10368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:19.013896 10368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:42063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:19.015043 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:19.016464 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:19.025559 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.025563 10377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.025538 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.025542 10368 server_base.cc:1034] running on GCE node
I20260806 02:41:19.026374 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:19.027405 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:19.028568 10368 hybrid_clock.cc:648] HybridClock initialized: now 1785984079028538 us; error 43 us; skew 500 ppm
I20260806 02:41:19.030471 10368 webserver.cc:460] Webserver started at http://127.28.191.62:35617/ using document root <none> and password file <none>
I20260806 02:41:19.031069 10368 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:19.031132 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:19.031415 10368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:19.033506 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "53e782538b5543d79c8fdac7a81b2757"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "f722eb7554c13989f99dfb919c9a42cb"
server_key_iv: "404ecfa7ad24d80c5227c337dffbe285"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.034075 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "53e782538b5543d79c8fdac7a81b2757"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "f722eb7554c13989f99dfb919c9a42cb"
server_key_iv: "404ecfa7ad24d80c5227c337dffbe285"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.037734 10368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:41:19.040017 10389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.041395 10368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 02:41:19.041553 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "53e782538b5543d79c8fdac7a81b2757"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "f722eb7554c13989f99dfb919c9a42cb"
server_key_iv: "404ecfa7ad24d80c5227c337dffbe285"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.041671 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:19.055687 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:19.056355 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:19.056557 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:19.063529 10368 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:42063
I20260806 02:41:19.063550 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:42063 every 8 connection(s)
I20260806 02:41:19.064445 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:41:19.067504 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 10368
I20260806 02:41:19.067648 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:41:19.067728 10459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:19.067972 29436 external_mini_cluster.cc:1428] Setting key dd08c15f7eeb13a3d3b7d1bbb6b068e1
I20260806 02:41:19.074622 10459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: Bootstrap starting.
I20260806 02:41:19.076819 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.069290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.28.191.62:42063)
0806 02:41:19.069851 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.069865 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.069899 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:41:19.070560 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.070576 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.070967 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.071303 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.071723 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.072663 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.073614 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.074073 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.074246 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.075173 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:19.075178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:19.075189 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:19.075554 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:19.075606 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:19.075625 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:19.075738 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:19.075889 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:41:19.076142 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260806 02:41:19.076841 10459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:19.077661 10459 log.cc:824] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:19.079654 10459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: No bootstrap required, opened a new log
I20260806 02:41:19.083005 10459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } }
I20260806 02:41:19.083258 10459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:19.083302 10459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e782538b5543d79c8fdac7a81b2757, State: Initialized, Role: FOLLOWER
I20260806 02:41:19.083707 10459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } }
I20260806 02:41:19.083838 10459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:41:19.083885 10459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:41:19.083956 10459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:19.085008 10459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } }
I20260806 02:41:19.085348 10459 leader_election.cc:302] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53e782538b5543d79c8fdac7a81b2757; no voters:
I20260806 02:41:19.085628 10459 leader_election.cc:288] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:41:19.085728 10470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:41:19.085968 10470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [term 1 LEADER]: Becoming Leader. State: Replica: 53e782538b5543d79c8fdac7a81b2757, State: Running, Role: LEADER
I20260806 02:41:19.086308 10470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } }
I20260806 02:41:19.086772 10459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:41:19.088011 10471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53e782538b5543d79c8fdac7a81b2757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } } }
I20260806 02:41:19.087994 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53e782538b5543d79c8fdac7a81b2757. Latest consensus state: current_term: 1 leader_uuid: "53e782538b5543d79c8fdac7a81b2757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e782538b5543d79c8fdac7a81b2757" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42063 } } }
I20260806 02:41:19.088220 10471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:19.088246 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757 [sys.catalog]: This master's current role is: LEADER
W20260806 02:41:19.091452 10495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:41:19.091539 10495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:41:19.091619 10486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:41:19.092386 10486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:41:19.097339 10486 catalog_manager.cc:1347] Generated new cluster ID: 94cf5384ba61480c8ff747ee1d1950df
I20260806 02:41:19.097404 10486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:41:19.109933 10486 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:41:19.111047 10486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:41:19.124852 10486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53e782538b5543d79c8fdac7a81b2757: Generated new TSK 0
I20260806 02:41:19.125741 10486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:41:19.133720 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--builtin_ntp_servers=127.28.191.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:41:19.248278 10500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:19.248533 10500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:19.248571 10500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:19.252625 10500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:19.252682 10500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:19.252733 10500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:41:19.257391 10500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:19.258503 10500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:19.259845 10500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:19.269588 10517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.269616 10500 server_base.cc:1034] running on GCE node
W20260806 02:41:19.269626 10512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.269644 10510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.270365 10500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:19.271026 10500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:19.272197 10500 hybrid_clock.cc:648] HybridClock initialized: now 1785984079272162 us; error 47 us; skew 500 ppm
I20260806 02:41:19.274075 10500 webserver.cc:460] Webserver started at http://127.28.191.1:40753/ using document root <none> and password file <none>
I20260806 02:41:19.274668 10500 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:19.274732 10500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:19.274997 10500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:19.277159 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "1fc25f24bfcc4d2f9ec88cd1690f44a0"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "89ab3521fa524e1202f0be1c5bb39e2f"
server_key_iv: "5e2ba5e66c68728058ef33bb627e8615"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.277755 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "1fc25f24bfcc4d2f9ec88cd1690f44a0"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "89ab3521fa524e1202f0be1c5bb39e2f"
server_key_iv: "5e2ba5e66c68728058ef33bb627e8615"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.281620 10500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:41:19.284160 10523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.285429 10500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:41:19.285564 10500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "1fc25f24bfcc4d2f9ec88cd1690f44a0"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "89ab3521fa524e1202f0be1c5bb39e2f"
server_key_iv: "5e2ba5e66c68728058ef33bb627e8615"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.285686 10500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:19.307979 10500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:19.308727 10500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:19.308965 10500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:19.309629 10500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:19.310637 10500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:19.310719 10500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.310799 10500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:19.310858 10500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.320688 10500 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:35315
I20260806 02:41:19.320744 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:35315 every 8 connection(s)
I20260806 02:41:19.321722 10500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:41:19.330766 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.323503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:46917 (local address 127.28.191.62:42063)
0806 02:41:19.323678 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.323683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.324711 (+ 1028us) server_negotiation.cc:407] Connection header received
0806 02:41:19.326143 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.326146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.326195 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.326278 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.327110 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.327635 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.328998 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.329234 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.329316 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.330358 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:19.330361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:19.330362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:19.330374 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:19.330391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:19.330397 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:19.330454 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:19.330603 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:41:19.330657 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:41:19.331489 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 10500
I20260806 02:41:19.331609 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:41:19.331599 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.323920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42063 (local address 127.28.191.1:46917)
0806 02:41:19.324569 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:19.324607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:19.325923 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:19.326434 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:19.326440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:19.326516 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:19.326983 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.326995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.327760 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.327764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.328887 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.328897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.329368 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.329371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.329504 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.330126 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:19.330128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:19.330166 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:19.330226 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:19.330471 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:19.330478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:19.330483 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:41:19.330586 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:41:19.330787 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260806 02:41:19.331938 29436 external_mini_cluster.cc:1428] Setting key a3811f0bd078643828da94367199b405
I20260806 02:41:19.332926 10683 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42063
I20260806 02:41:19.333216 10683 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:19.333736 10683 heartbeater.cc:511] Master 127.28.191.62:42063 requested a full tablet report, sending...
I20260806 02:41:19.333943 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--builtin_ntp_servers=127.28.191.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:41:19.335217 10408 ts_manager.cc:194] Registered new tserver with Master: 1fc25f24bfcc4d2f9ec88cd1690f44a0 (127.28.191.1:35315)
I20260806 02:41:19.336318 10408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.1:46917
W20260806 02:41:19.455883 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:19.456140 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:19.456176 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:19.460218 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:19.460268 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:19.460351 10686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:41:19.465003 10686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:19.466041 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:19.467381 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:19.476631 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.476677 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.476631 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.476970 10686 server_base.cc:1034] running on GCE node
I20260806 02:41:19.477460 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:19.478071 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:19.479264 10686 hybrid_clock.cc:648] HybridClock initialized: now 1785984079479244 us; error 32 us; skew 500 ppm
I20260806 02:41:19.481218 10686 webserver.cc:460] Webserver started at http://127.28.191.2:33337/ using document root <none> and password file <none>
I20260806 02:41:19.481798 10686 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:19.481896 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:19.482115 10686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:19.484143 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "47a85fbac68f4d14b5719aed536513d6"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "2b427d6133797bdd36e349d649fcacd0"
server_key_iv: "db23fdc7704a9c00df5641cc3f092418"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.484731 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "47a85fbac68f4d14b5719aed536513d6"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "2b427d6133797bdd36e349d649fcacd0"
server_key_iv: "db23fdc7704a9c00df5641cc3f092418"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.488662 10686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:41:19.491308 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.492483 10686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:41:19.492632 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "47a85fbac68f4d14b5719aed536513d6"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "2b427d6133797bdd36e349d649fcacd0"
server_key_iv: "db23fdc7704a9c00df5641cc3f092418"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.492748 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:19.509699 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:19.510341 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:19.510538 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:19.511118 10686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:19.512040 10686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:19.512111 10686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.512181 10686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:19.512287 10686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.520803 10686 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:43259
I20260806 02:41:19.520903 10870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:43259 every 8 connection(s)
I20260806 02:41:19.521724 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:41:19.530692 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.523546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:35049 (local address 127.28.191.62:42063)
0806 02:41:19.523685 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.523688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.524692 (+ 1004us) server_negotiation.cc:407] Connection header received
0806 02:41:19.526189 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.526192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.526252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.526352 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.527198 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.527727 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.528840 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.529103 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.529217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.530203 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:19.530207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:19.530209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:19.530228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:19.530246 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:19.530254 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:19.530328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:19.530495 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:41:19.530553 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:41:19.531438 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.523903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42063 (local address 127.28.191.2:35049)
0806 02:41:19.524542 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:19.524584 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:19.525969 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:19.526529 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:19.526536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:19.526615 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:19.527061 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.527072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.527864 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.527867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.528714 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.528723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.529256 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.529260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.529425 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.529955 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:19.529957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:19.529994 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:19.530055 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:19.530343 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:19.530355 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:19.530364 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:41:19.530480 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:41:19.530753 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260806 02:41:19.531893 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 10686
I20260806 02:41:19.532013 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:41:19.532415 29436 external_mini_cluster.cc:1428] Setting key 0168574b195351f71cc963fc63d686fa
I20260806 02:41:19.532611 10871 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42063
I20260806 02:41:19.532842 10871 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:19.533437 10871 heartbeater.cc:511] Master 127.28.191.62:42063 requested a full tablet report, sending...
I20260806 02:41:19.534440 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--builtin_ntp_servers=127.28.191.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:41:19.534458 10408 ts_manager.cc:194] Registered new tserver with Master: 47a85fbac68f4d14b5719aed536513d6 (127.28.191.2:43259)
I20260806 02:41:19.535012 10408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.2:35049
W20260806 02:41:19.648056 10875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:19.648293 10875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:19.648327 10875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:19.652290 10875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:19.652340 10875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:19.652390 10875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:41:19.656939 10875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:19.657914 10875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:19.659173 10875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:19.668666 10886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.668679 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.668660 10875 server_base.cc:1034] running on GCE node
W20260806 02:41:19.668646 10881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.669373 10875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:19.669973 10875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:19.671175 10875 hybrid_clock.cc:648] HybridClock initialized: now 1785984079671146 us; error 49 us; skew 500 ppm
I20260806 02:41:19.672870 10875 webserver.cc:460] Webserver started at http://127.28.191.3:46347/ using document root <none> and password file <none>
I20260806 02:41:19.673516 10875 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:19.673617 10875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:19.673825 10875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:19.675789 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "f02c1dc29a8e4251af89b7ca11360442"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "bffa18a5c51036ac0d9f0daba0aa55a3"
server_key_iv: "9feb22d502418bcdf3f8fc8b21995d4f"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.676379 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "f02c1dc29a8e4251af89b7ca11360442"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "bffa18a5c51036ac0d9f0daba0aa55a3"
server_key_iv: "9feb22d502418bcdf3f8fc8b21995d4f"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.679924 10875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:41:19.682524 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.683694 10875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:41:19.683830 10875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "f02c1dc29a8e4251af89b7ca11360442"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "bffa18a5c51036ac0d9f0daba0aa55a3"
server_key_iv: "9feb22d502418bcdf3f8fc8b21995d4f"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.683944 10875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:19.697908 10875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:19.698541 10875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:19.698737 10875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:19.699321 10875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:19.700266 10875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:19.700340 10875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.700412 10875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:19.700455 10875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.709131 10875 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:45695
I20260806 02:41:19.709159 11059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:45695 every 8 connection(s)
I20260806 02:41:19.709986 10875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:41:19.711563 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 10875
I20260806 02:41:19.711699 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:41:19.712023 29436 external_mini_cluster.cc:1428] Setting key 95d0328fef3a1c8627b527818a807f89
I20260806 02:41:19.718962 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.711721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60293 (local address 127.28.191.62:42063)
0806 02:41:19.711879 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.711885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.712980 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 02:41:19.714634 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.714636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.714683 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.714754 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.715611 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.716139 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.717310 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.717528 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.717602 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.718574 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:19.718577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:19.718579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:19.718591 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:19.718610 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:19.718618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:19.718675 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:19.718808 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:41:19.718856 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:41:19.719771 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.712078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42063 (local address 127.28.191.3:60293)
0806 02:41:19.712838 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:19.712891 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:19.714448 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:19.714916 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:19.714923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:19.715005 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:19.715490 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.715502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.716246 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.716248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.717202 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:19.717212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:19.717652 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:19.717655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:19.717792 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:19.718330 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:19.718332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:19.718370 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:19.718431 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:19.718690 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:19.718696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:19.718701 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:41:19.718791 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:41:19.718998 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260806 02:41:19.720849 11060 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42063
I20260806 02:41:19.721099 11060 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:19.721612 11060 heartbeater.cc:511] Master 127.28.191.62:42063 requested a full tablet report, sending...
I20260806 02:41:19.722621 10408 ts_manager.cc:194] Registered new tserver with Master: f02c1dc29a8e4251af89b7ca11360442 (127.28.191.3:45695)
I20260806 02:41:19.723170 10408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.3:60293
I20260806 02:41:19.726150 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:41:19.729017 10462 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:19.728044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.28.191.62:42063)
0806 02:41:19.728213 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.728217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.728378 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:41:19.728526 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.728530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.728589 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.728677 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.728933 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57580: BlockingRecv error: recv got EOF from 127.0.0.1:57580 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:41:19.729524 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 10500
I20260806 02:41:19.738385 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 10686
I20260806 02:41:19.751451 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 10875
I20260806 02:41:19.760526 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 10368
2026-08-06T02:41:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (911 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:19Z Disabled control of system clock
I20260806 02:41:19.792585 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35487
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:41:19.903524 11078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:19.903772 11078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:19.903808 11078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:19.907611 11078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:19.907660 11078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:19.907681 11078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:19.907698 11078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:19.907713 11078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:19.912168 11078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:19.913209 11078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:19.914471 11078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:19.922605 11094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:19.922643 11088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.922683 11078 server_base.cc:1034] running on GCE node
W20260806 02:41:19.922659 11087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:19.923378 11078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:19.924445 11078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:19.925638 11078 hybrid_clock.cc:648] HybridClock initialized: now 1785984079925607 us; error 52 us; skew 500 ppm
I20260806 02:41:19.927450 11078 webserver.cc:460] Webserver started at http://127.28.191.62:35353/ using document root <none> and password file <none>
I20260806 02:41:19.928054 11078 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:19.928153 11078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:19.928391 11078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:19.930517 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "d6744fb345114f59abfa331880609a6c"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "801dcda3bbc977e8f2c8c3988ec4866e"
server_key_iv: "5a7f13a64278cd34c96d5dd0c9364aef"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.931107 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "d6744fb345114f59abfa331880609a6c"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "801dcda3bbc977e8f2c8c3988ec4866e"
server_key_iv: "5a7f13a64278cd34c96d5dd0c9364aef"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.934779 11078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:41:19.937294 11104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:19.938468 11078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:41:19.938623 11078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "d6744fb345114f59abfa331880609a6c"
format_stamp: "Formatted at 2026-08-06 02:41:19 on dist-test-slave-tfpf"
server_key: "801dcda3bbc977e8f2c8c3988ec4866e"
server_key_iv: "5a7f13a64278cd34c96d5dd0c9364aef"
server_key_version: "encryptionkey@0"
I20260806 02:41:19.938740 11078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:19.943243 11078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:19.943835 11078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:19.944018 11078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:19.950829 11078 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35487
I20260806 02:41:19.950835 11182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35487 every 8 connection(s)
I20260806 02:41:19.951715 11078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:41:19.955010 11184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:19.958258 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11078
I20260806 02:41:19.958390 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:41:19.958702 29436 external_mini_cluster.cc:1428] Setting key aa37e78991e35dc2d8e2e9b2a4eeac44
I20260806 02:41:19.960954 11184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: Bootstrap starting.
I20260806 02:41:19.963217 11184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:19.963990 11184 log.cc:824] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:19.964200 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:19.960109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.28.191.62:35487)
0806 02:41:19.960708 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:19.960722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:19.960755 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:41:19.961471 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:19.961491 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:19.961821 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:19.962160 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:19.962970 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:19.962974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:19.962986 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:19.962996 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:19.963053 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:19.963070 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:19.963191 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:19.963302 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 02:41:19.963526 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":130,"threads_started":1}
I20260806 02:41:19.966068 11184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: No bootstrap required, opened a new log
I20260806 02:41:19.969450 11184 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } }
I20260806 02:41:19.969674 11184 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:19.969754 11184 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6744fb345114f59abfa331880609a6c, State: Initialized, Role: FOLLOWER
I20260806 02:41:19.970197 11184 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } }
I20260806 02:41:19.970345 11184 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:41:19.970430 11184 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:41:19.970528 11184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:19.971331 11184 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } }
I20260806 02:41:19.971662 11184 leader_election.cc:302] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6744fb345114f59abfa331880609a6c; no voters:
I20260806 02:41:19.971948 11184 leader_election.cc:288] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:41:19.972077 11190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:41:19.972318 11190 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [term 1 LEADER]: Becoming Leader. State: Replica: d6744fb345114f59abfa331880609a6c, State: Running, Role: LEADER
I20260806 02:41:19.972685 11190 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } }
I20260806 02:41:19.973160 11184 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:41:19.974274 11192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6744fb345114f59abfa331880609a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } } }
I20260806 02:41:19.974416 11192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:19.974799 11193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6744fb345114f59abfa331880609a6c. Latest consensus state: current_term: 1 leader_uuid: "d6744fb345114f59abfa331880609a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744fb345114f59abfa331880609a6c" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35487 } } }
I20260806 02:41:19.975487 11193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c [sys.catalog]: This master's current role is: LEADER
W20260806 02:41:19.977850 11214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:41:19.977916 11214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:41:19.977994 11202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:41:19.978674 11202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:41:19.983569 11202 catalog_manager.cc:1347] Generated new cluster ID: 406583476f634d669796a7be9010a9ef
I20260806 02:41:19.983630 11202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:41:19.993114 11202 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:41:19.993475 11202 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:41:20.005585 11202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6744fb345114f59abfa331880609a6c: Generated new TSK 0
I20260806 02:41:20.006510 11202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:41:20.011422 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--builtin_ntp_servers=127.28.191.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:41:20.125839 11218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:20.126138 11218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:20.126179 11218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:20.130182 11218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:20.130232 11218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:20.130285 11218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:41:20.134857 11218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:20.135815 11218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:20.137172 11218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:20.146983 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:20.146983 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:20.147019 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:20.147254 11218 server_base.cc:1034] running on GCE node
I20260806 02:41:20.147846 11218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:20.148561 11218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:20.149737 11218 hybrid_clock.cc:648] HybridClock initialized: now 1785984080149715 us; error 34 us; skew 500 ppm
I20260806 02:41:20.151821 11218 webserver.cc:460] Webserver started at http://127.28.191.1:32961/ using document root <none> and password file <none>
I20260806 02:41:20.152432 11218 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:20.152534 11218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:20.152793 11218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:20.155135 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "f994d82759144d3c8e358393739dfb36"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "f317d494f6858b59f6e3fc4236ffd792"
server_key_iv: "769b9a1a08178b1952e24e6869af342c"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.155754 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "f994d82759144d3c8e358393739dfb36"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "f317d494f6858b59f6e3fc4236ffd792"
server_key_iv: "769b9a1a08178b1952e24e6869af342c"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.159816 11218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:41:20.162447 11235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.163664 11218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:41:20.163818 11218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f994d82759144d3c8e358393739dfb36"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "f317d494f6858b59f6e3fc4236ffd792"
server_key_iv: "769b9a1a08178b1952e24e6869af342c"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.163938 11218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:20.168953 11218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:20.169654 11218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:20.169855 11218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:20.170413 11218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:20.171355 11218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:20.171433 11218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.171527 11218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:20.171589 11218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.181218 11218 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:41485
I20260806 02:41:20.181246 11402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:41485 every 8 connection(s)
I20260806 02:41:20.182082 11218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:41:20.187973 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.183841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:55909 (local address 127.28.191.62:35487)
0806 02:41:20.184001 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:20.184006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:20.184982 (+ 976us) server_negotiation.cc:407] Connection header received
0806 02:41:20.186395 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:20.186398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:20.186434 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:20.186507 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:20.187639 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:20.187641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:20.187643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:20.187652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:20.187666 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:20.187673 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:20.187740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:20.187836 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:41:20.187872 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:41:20.188791 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11218
I20260806 02:41:20.188933 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:41:20.189028 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.184198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35487 (local address 127.28.191.1:55909)
0806 02:41:20.184840 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:20.184881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:20.186208 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:20.186640 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:20.186647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:20.186732 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:20.187381 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:20.187383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:20.187424 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:20.187503 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:20.187729 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:20.187733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:20.187737 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:41:20.187861 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:41:20.188093 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260806 02:41:20.189347 29436 external_mini_cluster.cc:1428] Setting key d93dfebedcafa173dcc9d6681cd5fdb8
I20260806 02:41:20.190361 11403 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35487
I20260806 02:41:20.190599 11403 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:20.191006 11403 heartbeater.cc:511] Master 127.28.191.62:35487 requested a full tablet report, sending...
I20260806 02:41:20.191463 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--builtin_ntp_servers=127.28.191.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:41:20.192705 11128 ts_manager.cc:194] Registered new tserver with Master: f994d82759144d3c8e358393739dfb36 (127.28.191.1:41485)
W20260806 02:41:20.304257 11408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:20.304574 11408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:20.304649 11408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:20.308657 11408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:20.308733 11408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:20.308826 11408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:41:20.313476 11408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:20.314539 11408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:20.315878 11408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:20.325037 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:20.325047 11416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:20.325148 11408 server_base.cc:1034] running on GCE node
W20260806 02:41:20.325047 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:20.325713 11408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:20.326314 11408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:20.327541 11408 hybrid_clock.cc:648] HybridClock initialized: now 1785984080327525 us; error 26 us; skew 500 ppm
I20260806 02:41:20.329653 11408 webserver.cc:460] Webserver started at http://127.28.191.2:40971/ using document root <none> and password file <none>
I20260806 02:41:20.330235 11408 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:20.330305 11408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:20.330588 11408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:20.332624 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "c98a2e69107f4ef0b79296dda7d8da43"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "9b6527ceed46fd04ce871ecb82bfc84e"
server_key_iv: "f14dd81f8a0ba42879b851cdb9f8e5c8"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.333278 11408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "c98a2e69107f4ef0b79296dda7d8da43"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "9b6527ceed46fd04ce871ecb82bfc84e"
server_key_iv: "f14dd81f8a0ba42879b851cdb9f8e5c8"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.336982 11408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:41:20.339551 11431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.340767 11408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:41:20.340975 11408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "c98a2e69107f4ef0b79296dda7d8da43"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "9b6527ceed46fd04ce871ecb82bfc84e"
server_key_iv: "f14dd81f8a0ba42879b851cdb9f8e5c8"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.341130 11408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:20.345815 11408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:20.346391 11408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:20.346577 11408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:20.347102 11408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:20.347980 11408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:20.348053 11408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.348120 11408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:20.348168 11408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.356984 11408 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:33523
I20260806 02:41:20.357033 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:33523 every 8 connection(s)
I20260806 02:41:20.357890 11408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:41:20.358732 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11408
I20260806 02:41:20.358871 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:41:20.359150 29436 external_mini_cluster.cc:1428] Setting key b14f0de4c76cd72ee4ad34e1a895e264
I20260806 02:41:20.361313 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--builtin_ntp_servers=127.28.191.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:41:20.364265 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.359797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:47609 (local address 127.28.191.62:35487)
0806 02:41:20.359950 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:20.359955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:20.361106 (+ 1151us) server_negotiation.cc:407] Connection header received
0806 02:41:20.362542 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:20.362546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:20.362589 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:20.362669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:20.363758 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:20.363762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:20.363765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:20.363777 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:20.363799 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:20.363815 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:20.363903 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:20.364089 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:41:20.364135 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:41:20.365135 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.360221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35487 (local address 127.28.191.2:47609)
0806 02:41:20.360933 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:20.360971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:20.362329 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:20.362828 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:20.362834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:20.362919 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:20.363503 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:20.363504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:20.363546 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:20.363610 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:20.363901 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:20.363911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:20.363919 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:41:20.364126 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:41:20.364422 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":147,"threads_started":1}
I20260806 02:41:20.366246 11587 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35487
I20260806 02:41:20.366474 11587 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:20.366869 11587 heartbeater.cc:511] Master 127.28.191.62:35487 requested a full tablet report, sending...
I20260806 02:41:20.367925 11128 ts_manager.cc:194] Registered new tserver with Master: c98a2e69107f4ef0b79296dda7d8da43 (127.28.191.2:33523)
W20260806 02:41:20.474910 11593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:20.475186 11593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:20.475271 11593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:20.479336 11593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:20.479408 11593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:20.479501 11593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:41:20.484131 11593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:20.485172 11593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:20.486495 11593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:20.495798 11599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:20.495790 11598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:20.495905 11593 server_base.cc:1034] running on GCE node
W20260806 02:41:20.495790 11601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:20.496461 11593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:20.497030 11593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:20.498284 11593 hybrid_clock.cc:648] HybridClock initialized: now 1785984080498270 us; error 25 us; skew 500 ppm
I20260806 02:41:20.504707 11593 webserver.cc:460] Webserver started at http://127.28.191.3:36093/ using document root <none> and password file <none>
I20260806 02:41:20.505430 11593 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:20.505529 11593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:20.505801 11593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:20.507861 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "9b40f5421b714b72b39a52c2c3a54abb"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "44d4d3be0d6c05c4a6416823cc164ff6"
server_key_iv: "68308e1ae0ff167e5eeac57802ceefb0"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.508455 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "9b40f5421b714b72b39a52c2c3a54abb"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "44d4d3be0d6c05c4a6416823cc164ff6"
server_key_iv: "68308e1ae0ff167e5eeac57802ceefb0"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.512091 11593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260806 02:41:20.514768 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.515889 11593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:41:20.516023 11593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "9b40f5421b714b72b39a52c2c3a54abb"
format_stamp: "Formatted at 2026-08-06 02:41:20 on dist-test-slave-tfpf"
server_key: "44d4d3be0d6c05c4a6416823cc164ff6"
server_key_iv: "68308e1ae0ff167e5eeac57802ceefb0"
server_key_version: "encryptionkey@0"
I20260806 02:41:20.516153 11593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:20.520620 11593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:20.521289 11593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:20.521461 11593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:20.521978 11593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:20.522851 11593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:20.522922 11593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.522991 11593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:20.523051 11593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:20.531505 11593 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:43573
I20260806 02:41:20.531539 11759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:43573 every 8 connection(s)
I20260806 02:41:20.532330 11593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:41:20.537998 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.534124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:36339 (local address 127.28.191.62:35487)
0806 02:41:20.534264 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:20.534267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:20.535220 (+ 953us) server_negotiation.cc:407] Connection header received
0806 02:41:20.536429 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:20.536431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:20.536470 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:20.536546 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:20.537632 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:20.537635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:20.537637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:41:20.537646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:20.537662 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:41:20.537669 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:20.537740 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:20.537861 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:41:20.537898 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:41:20.538786 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:20.534428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:35487 (local address 127.28.191.3:36339)
0806 02:41:20.535060 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:20.535097 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:20.536228 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:20.536692 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:20.536699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:20.536793 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:20.537399 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:41:20.537401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:20.537437 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:41:20.537500 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:20.537744 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:20.537750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:20.537754 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:41:20.537875 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:41:20.538106 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":143,"threads_started":1}
I20260806 02:41:20.539215 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11593
I20260806 02:41:20.539321 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:41:20.539646 29436 external_mini_cluster.cc:1428] Setting key 6efef99427462fee8c6b4209e63c65dc
I20260806 02:41:20.539963 11761 heartbeater.cc:347] Connected to a master server at 127.28.191.62:35487
I20260806 02:41:20.540186 11761 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:20.540589 11761 heartbeater.cc:511] Master 127.28.191.62:35487 requested a full tablet report, sending...
I20260806 02:41:20.541600 11126 ts_manager.cc:194] Registered new tserver with Master: 9b40f5421b714b72b39a52c2c3a54abb (127.28.191.3:43573)
I20260806 02:41:20.553300 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:41:20.556288 11188 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:41:20.555301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.28.191.62:35487)
0806 02:41:20.555419 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:20.555423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:20.555665 (+ 242us) server_negotiation.cc:407] Connection header received
0806 02:41:20.555815 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:20.555821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:20.555859 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:20.555942 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:20.556191 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41078: BlockingRecv error: recv got EOF from 127.0.0.1:41078 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:41:20.556684 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11218
I20260806 02:41:20.573040 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11408
I20260806 02:41:20.581714 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11593
I20260806 02:41:20.590092 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11078
2026-08-06T02:41:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (834 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:41:20 dist-test-slave-tfpf krb5kdc[11778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:41:20 dist-test-slave-tfpf krb5kdc[11778](info): set up 2 sockets
Aug 06 02:41:20 dist-test-slave-tfpf krb5kdc[11778](info): commencing operation
krb5kdc: starting...
W20260806 02:41:21.839920 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:41:21 dist-test-slave-tfpf krb5kdc[11778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:41:21.985018 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:41227
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:22.187167 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:22.187559 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:22.187639 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:22.193989 11796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:41:22.194129 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:22.194152 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:22.194180 11796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:41:22.194265 11796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:41:22.202549 11796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:22.204397 11796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:22.206331 11796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:22.220530 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:22.220587 11805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:22.220587 11807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:22.221318 11796 server_base.cc:1034] running on GCE node
I20260806 02:41:22.222191 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:22.223939 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:22.225225 11796 hybrid_clock.cc:648] HybridClock initialized: now 1785984082225189 us; error 52 us; skew 500 ppm
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984082, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:22.229837 11796 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:41:22.231426 11796 webserver.cc:460] Webserver started at http://127.28.191.62:41437/ using document root <none> and password file <none>
I20260806 02:41:22.232348 11796 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:22.232465 11796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:22.232914 11796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:22.236508 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "8e2f8ec17ccb4ba5be97e965fd432195"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d0ee7d4d1561d07fdf1120fe4dc2f04e"
server_key_iv: "90a29c0fc42a8d8784877340770c2f2d"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.237668 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "8e2f8ec17ccb4ba5be97e965fd432195"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d0ee7d4d1561d07fdf1120fe4dc2f04e"
server_key_iv: "90a29c0fc42a8d8784877340770c2f2d"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.244478 11796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:41:22.248780 11817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.251097 11796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:41:22.251324 11796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "8e2f8ec17ccb4ba5be97e965fd432195"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d0ee7d4d1561d07fdf1120fe4dc2f04e"
server_key_iv: "90a29c0fc42a8d8784877340770c2f2d"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.251523 11796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:22.276458 11796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:22.281785 11796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:22.282166 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:22.295509 11796 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:41227
I20260806 02:41:22.295998 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:41227 every 8 connection(s)
I20260806 02:41:22.297281 11796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:41:22.302194 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:22.307158 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11796
I20260806 02:41:22.307381 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:41:22.307981 29436 external_mini_cluster.cc:1428] Setting key fac457673f4bfa55f53b0ad467e8da64
I20260806 02:41:22.313961 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: Bootstrap starting.
I20260806 02:41:22.317725 11885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:22.319044 11885 log.cc:824] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:22.323511 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: No bootstrap required, opened a new log
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:22.332161 11885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } }
I20260806 02:41:22.332826 11885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:22.332962 11885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2f8ec17ccb4ba5be97e965fd432195, State: Initialized, Role: FOLLOWER
I20260806 02:41:22.333845 11885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } }
I20260806 02:41:22.334113 11885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:41:22.334259 11885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:41:22.334486 11885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:22.336354 11885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } }
I20260806 02:41:22.336943 11885 leader_election.cc:302] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2f8ec17ccb4ba5be97e965fd432195; no voters:
I20260806 02:41:22.337252 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:22.310931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.28.191.62:41227)
0806 02:41:22.311653 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:22.311677 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:22.311748 (+ 71us) server_negotiation.cc:407] Connection header received
0806 02:41:22.312870 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:22.312901 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:22.313590 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:22.314170 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:22.315772 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.317412 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.319037 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.320219 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.320490 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:22.325839 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:22.325867 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:22.325906 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:22.325991 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:22.331491 (+ 5500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.332307 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.332325 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.332331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.332614 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.333234 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.333252 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.333255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.334127 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:22.334604 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:22.335163 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 02:41:22.335651 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":225,"threads_started":1}
I20260806 02:41:22.337524 11885 leader_election.cc:288] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:41:22.337718 11891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:41:22.338061 11891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2f8ec17ccb4ba5be97e965fd432195, State: Running, Role: LEADER
I20260806 02:41:22.338632 11891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } }
I20260806 02:41:22.339912 11885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:41:22.341836 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } } }
I20260806 02:41:22.342114 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:22.342748 11908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:41:22.342728 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e2f8ec17ccb4ba5be97e965fd432195. Latest consensus state: current_term: 1 leader_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8ec17ccb4ba5be97e965fd432195" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 41227 } } }
I20260806 02:41:22.342921 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195 [sys.catalog]: This master's current role is: LEADER
I20260806 02:41:22.347988 11908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:41:22.357697 11908 catalog_manager.cc:1347] Generated new cluster ID: 0d23e569b50a4e9088782ccc1c3cfc49
I20260806 02:41:22.357831 11908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:41:22.399106 11908 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:41:22.400684 11908 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:41:22.412930 11908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: Generated new TSK 0
I20260806 02:41:22.414122 11908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:41:22.473182 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:22.625514 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:22.625753 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:22.625797 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:22.629698 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:22.629747 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:22.629803 11926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:41:22.634943 11926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:22.635859 11926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:22.636790 11926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:22.646431 11937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:22.646490 11934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:22.646522 11926 server_base.cc:1034] running on GCE node
W20260806 02:41:22.646682 11935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:22.647118 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:22.647722 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:22.648912 11926 hybrid_clock.cc:648] HybridClock initialized: now 1785984082648881 us; error 43 us; skew 500 ppm
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984082, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:22.651540 11926 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:41:22.652390 11926 webserver.cc:460] Webserver started at http://127.28.191.1:33435/ using document root <none> and password file <none>
I20260806 02:41:22.652932 11926 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:22.653024 11926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:22.653276 11926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:22.655257 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "38aa71a69a1048c3b02612b9f27e61b6"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d7843565b872f79da78466ac7789c402"
server_key_iv: "f7c798c85834468425df4a7a64061d6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.655829 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "38aa71a69a1048c3b02612b9f27e61b6"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d7843565b872f79da78466ac7789c402"
server_key_iv: "f7c798c85834468425df4a7a64061d6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.659664 11926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:41:22.662048 11944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.663210 11926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:41:22.663352 11926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "38aa71a69a1048c3b02612b9f27e61b6"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "d7843565b872f79da78466ac7789c402"
server_key_iv: "f7c798c85834468425df4a7a64061d6e"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.663458 11926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:22.686421 11926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:22.689184 11926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:22.689378 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:22.689953 11926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:22.690812 11926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:22.690884 11926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.690950 11926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:22.691004 11926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.699774 11926 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42359
I20260806 02:41:22.699800 12121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42359 every 8 connection(s)
I20260806 02:41:22.700605 11926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:41:22.704385 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 11926
I20260806 02:41:22.704484 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:41:22.704761 29436 external_mini_cluster.cc:1428] Setting key fdae1f4f9258ddb78dae4c865da3ee28
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984082, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:22.714852 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:22.702361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:37109 (local address 127.28.191.62:41227)
0806 02:41:22.702492 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:22.702496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:22.703220 (+ 724us) server_negotiation.cc:407] Connection header received
0806 02:41:22.704564 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:22.704568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:22.704631 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:22.704743 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:22.706185 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.706872 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.707984 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.708220 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.708336 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:22.711080 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:22.711083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:22.711085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:22.711107 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:22.712542 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.713247 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.713250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.713252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.713333 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.713739 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.713742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.713743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.713915 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:22.713990 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:22.714625 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 02:41:22.714731 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:41:22.716028 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:22.702633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41227 (local address 127.28.191.1:37109)
0806 02:41:22.703050 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:22.703090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:22.704311 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:22.704921 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:22.704928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:22.705369 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:22.706010 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:22.706051 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.707000 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.707003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:22.707853 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:22.707863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.708372 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.708376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:22.708535 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:22.709169 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:22.709191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:22.710919 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:22.712666 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:22.712675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:22.712691 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:22.713117 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:22.713433 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:22.713439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:22.713441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:22.713630 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:22.714022 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:22.714031 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:22.714355 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 02:41:22.714572 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 02:41:22.714915 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260806 02:41:22.717382 12124 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41227
I20260806 02:41:22.717607 12124 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:22.718137 12124 heartbeater.cc:511] Master 127.28.191.62:41227 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:41:22.719786 11837 ts_manager.cc:194] Registered new tserver with Master: 38aa71a69a1048c3b02612b9f27e61b6 (127.28.191.1:42359)
I20260806 02:41:22.720913 11837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:37109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:41:22.733178 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:22.846132 12130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:22.846390 12130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:22.846465 12130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:22.850517 12130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:22.850574 12130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:22.850688 12130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:41:22.859576 12130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:22.861025 12130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:22.862438 12130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:22.873647 12139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:22.873696 12137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:22.873842 12130 server_base.cc:1034] running on GCE node
W20260806 02:41:22.873683 12136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:22.874483 12130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:22.875104 12130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:22.876272 12130 hybrid_clock.cc:648] HybridClock initialized: now 1785984082876253 us; error 33 us; skew 500 ppm
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984082, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:22.879161 12130 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:41:22.880089 12130 webserver.cc:460] Webserver started at http://127.28.191.2:40017/ using document root <none> and password file <none>
I20260806 02:41:22.880654 12130 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:22.880717 12130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:22.880966 12130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:22.883054 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "6e2c0d9bda5a5a1fd597c2cf37ad54f5"
server_key_iv: "c6663aacacfe56545d9ba8da42d2cde0"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.883620 12130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "6e2c0d9bda5a5a1fd597c2cf37ad54f5"
server_key_iv: "c6663aacacfe56545d9ba8da42d2cde0"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.887446 12130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:41:22.889873 12148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.891068 12130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:41:22.891211 12130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9"
format_stamp: "Formatted at 2026-08-06 02:41:22 on dist-test-slave-tfpf"
server_key: "6e2c0d9bda5a5a1fd597c2cf37ad54f5"
server_key_iv: "c6663aacacfe56545d9ba8da42d2cde0"
server_key_version: "encryptionkey@0"
I20260806 02:41:22.891319 12130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:22.901546 12130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:22.904158 12130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:22.904340 12130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:22.904871 12130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:22.905709 12130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:22.905781 12130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.905846 12130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:22.905895 12130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:22.914942 12130 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:33817
I20260806 02:41:22.914963 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:33817 every 8 connection(s)
I20260806 02:41:22.915824 12130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:41:22.919335 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 12130
I20260806 02:41:22.919446 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:41:22.919724 29436 external_mini_cluster.cc:1428] Setting key 440627b1f0707035ffbde8e51d877edf
Aug 06 02:41:22 dist-test-slave-tfpf krb5kdc[11778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984082, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:22.929893 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:22.917578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:52733 (local address 127.28.191.62:41227)
0806 02:41:22.917702 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:22.917706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:22.918505 (+ 799us) server_negotiation.cc:407] Connection header received
0806 02:41:22.920124 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:22.920128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:22.920185 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:22.920286 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:22.921471 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.921991 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.923192 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.923454 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.923540 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:22.926315 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:22.926319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:22.926321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:22.926347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:22.927821 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.928342 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.928346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.928348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.928431 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:22.928777 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:22.928780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:22.928782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:22.928962 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:22.929096 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:22.929598 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 02:41:22.929723 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:41:22.930694 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:22.917922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41227 (local address 127.28.191.2:52733)
0806 02:41:22.918353 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:22.918393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:22.919919 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:22.920450 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:22.920456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:22.920802 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:22.921332 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:22.921346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.922124 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.922145 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:22.923025 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:22.923040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:22.923609 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:22.923615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:22.923835 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:22.924452 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:22.924477 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:22.926148 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:22.927946 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:22.927952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:22.927966 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:22.928233 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:22.928526 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:22.928531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:22.928533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:22.928670 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:22.929108 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:22.929116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:22.929350 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:41:22.929543 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 02:41:22.929847 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260806 02:41:22.931782 12322 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41227
I20260806 02:41:22.931993 12322 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:22.932435 12322 heartbeater.cc:511] Master 127.28.191.62:41227 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
I20260806 02:41:22.933472 11837 ts_manager.cc:194] Registered new tserver with Master: e8e5bd0a455a4f2e80c8028ca2b5fcf9 (127.28.191.2:33817)
I20260806 02:41:22.934321 11837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:52733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:41:22.947903 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:41:23.064129 12327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:41:23.064404 12327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:41:23.064477 12327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:41:23.068647 12327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:41:23.068727 12327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:41:23.068812 12327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:41:23.074113 12327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:41:23.075109 12327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:41:23.076082 12327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:41:23.085558 12343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:41:23.085557 12335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:23.085896 12327 server_base.cc:1034] running on GCE node
W20260806 02:41:23.085570 12336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:41:23.086310 12327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:41:23.086855 12327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:41:23.088012 12327 hybrid_clock.cc:648] HybridClock initialized: now 1785984083087985 us; error 41 us; skew 500 ppm
Aug 06 02:41:23 dist-test-slave-tfpf krb5kdc[11778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:41:23.090668 12327 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:41:23.091478 12327 webserver.cc:460] Webserver started at http://127.28.191.3:33015/ using document root <none> and password file <none>
I20260806 02:41:23.092015 12327 fs_manager.cc:359] Metadata directory not provided
I20260806 02:41:23.092119 12327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:41:23.092345 12327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:41:23.094429 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "bfe2d977ae8e44dc93fb3993d2c7717a"
format_stamp: "Formatted at 2026-08-06 02:41:23 on dist-test-slave-tfpf"
server_key: "85e3659b319b6f4b06b943409731f5b5"
server_key_iv: "814e297073d26718d02691dbd9c7e19e"
server_key_version: "encryptionkey@0"
I20260806 02:41:23.094981 12327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe2d977ae8e44dc93fb3993d2c7717a"
format_stamp: "Formatted at 2026-08-06 02:41:23 on dist-test-slave-tfpf"
server_key: "85e3659b319b6f4b06b943409731f5b5"
server_key_iv: "814e297073d26718d02691dbd9c7e19e"
server_key_version: "encryptionkey@0"
I20260806 02:41:23.098613 12327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 02:41:23.100836 12352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:23.101995 12327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:41:23.102128 12327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "bfe2d977ae8e44dc93fb3993d2c7717a"
format_stamp: "Formatted at 2026-08-06 02:41:23 on dist-test-slave-tfpf"
server_key: "85e3659b319b6f4b06b943409731f5b5"
server_key_iv: "814e297073d26718d02691dbd9c7e19e"
server_key_version: "encryptionkey@0"
I20260806 02:41:23.102236 12327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:41:23.116914 12327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:41:23.119699 12327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:41:23.119890 12327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:41:23.120447 12327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:41:23.121309 12327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:41:23.121379 12327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:23.121452 12327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:41:23.121495 12327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:41:23.130442 12327 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36873
I20260806 02:41:23.130477 12526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36873 every 8 connection(s)
I20260806 02:41:23.131325 12327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:41:23.134111 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 12327
I20260806 02:41:23.134229 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:41:23.134591 29436 external_mini_cluster.cc:1428] Setting key afc94fb11bb145612c93696abd1bdf9f
Aug 06 02:41:23 dist-test-slave-tfpf krb5kdc[11778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984083, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:41:23.144740 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.133118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42139 (local address 127.28.191.62:41227)
0806 02:41:23.133253 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.133257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.134095 (+ 838us) server_negotiation.cc:407] Connection header received
0806 02:41:23.135722 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.135725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.135792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.135877 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:23.137239 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.137757 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.138825 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.139034 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.139115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.141562 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:23.141602 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:23.141605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:23.141637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:23.143003 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:23.143499 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:23.143502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:23.143503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:23.143578 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:23.143869 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:23.143871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:23.143872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:23.144029 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:23.144119 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.144533 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.144626 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:41:23.145519 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.133452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:41227 (local address 127.28.191.3:42139)
0806 02:41:23.133898 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.133956 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.135486 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.136100 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.136114 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.136457 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:41:23.137054 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.137129 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.137859 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.137862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.138718 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.138728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.139161 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.139163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.139298 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.139889 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:41:23.139905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:41:23.141411 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:41:23.143126 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:23.143130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:23.143140 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:23.143396 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:23.143651 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:41:23.143653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:41:23.143655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:41:23.143782 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:41:23.144129 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:41:23.144133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:41:23.144334 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.144472 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.144696 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260806 02:41:23.146552 12527 heartbeater.cc:347] Connected to a master server at 127.28.191.62:41227
I20260806 02:41:23.146745 12527 heartbeater.cc:464] Registering TS with master...
I20260806 02:41:23.147276 12527 heartbeater.cc:511] Master 127.28.191.62:41227 requested a full tablet report, sending...
I20260806 02:41:23.148319 11835 ts_manager.cc:194] Registered new tserver with Master: bfe2d977ae8e44dc93fb3993d2c7717a (127.28.191.3:36873)
I20260806 02:41:23.148834 11835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:42139
I20260806 02:41:23.159781 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:41:23.168534 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.161715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.28.191.62:41227)
0806 02:41:23.161846 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.161850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.162022 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:41:23.162133 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.162136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.162171 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.162233 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:41:23.162855 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.163337 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.164127 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.164320 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.164390 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.165739 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:41:23.165741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:41:23.165743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:41:23.165761 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:41:23.167036 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:23.167399 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:23.167402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:23.167403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:23.167472 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:41:23.167701 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:41:23.167703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:41:23.167704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:41:23.167943 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:41:23.168046 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.168250 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.168337 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:41:23.171648 11835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:41:23.174605 11835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:41:23.188921 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.182681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:33817 (local address 127.0.0.1:57280)
0806 02:41:23.183904 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.183918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.184046 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.184471 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.184474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.184492 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:41:23.184584 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.184591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.185766 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.185771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.187326 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.187337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.188438 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.188441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.188517 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.188548 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.188656 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.188770 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":87,"thread_start_us":112,"threads_started":1}
I20260806 02:41:23.188989 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.183274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.28.191.2:33817)
0806 02:41:23.183761 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.183774 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.184118 (+ 344us) server_negotiation.cc:407] Connection header received
0806 02:41:23.184205 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.184209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.184335 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.184488 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:41:23.184699 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.185646 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.187463 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.188313 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.188449 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.188627 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.188701 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.188811 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":116,"threads_started":1}
I20260806 02:41:23.189275 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.183267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.28.191.1:42359)
0806 02:41:23.184589 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.184608 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.184630 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:41:23.184728 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.184733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.184908 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.185110 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:41:23.185376 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.186386 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.187712 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.188503 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.188617 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.188749 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.189000 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.189143 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":75,"threads_started":1}
I20260806 02:41:23.189459 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.183121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42359 (local address 127.0.0.1:58330)
0806 02:41:23.184249 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.184264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.184341 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.185113 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.185116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.185135 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:41:23.185230 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.185235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.186544 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.186547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.187536 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.187548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.188625 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.188631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.188726 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.188747 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.188824 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.188879 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":90,"threads_started":1}
I20260806 02:41:23.190516 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.182493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:36873 (local address 127.0.0.1:50772)
0806 02:41:23.183355 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.183408 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.183521 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.184281 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.184291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.184318 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:41:23.184482 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.184490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.185976 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.185980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.187326 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.187337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.190059 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.190064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.190168 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.190201 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.190281 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.190366 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":80,"spinlock_wait_cycles":896,"thread_start_us":68,"threads_started":1}
I20260806 02:41:23.190964 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.182676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.28.191.3:36873)
0806 02:41:23.183485 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.183499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.183532 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:41:23.183711 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.183718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.183921 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.184081 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:41:23.184628 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.185830 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.188828 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.189703 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.189822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.189958 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.190702 (+ 744us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.190803 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":165,"threads_started":1}
I20260806 02:41:23.192457 12014 tablet_service.cc:1467] Processing CreateTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 (DEFAULT_TABLE table=test-table [id=d2c9a7703de24e4a8ab28f0f223dcbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:41:23.192504 12225 tablet_service.cc:1467] Processing CreateTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 (DEFAULT_TABLE table=test-table [id=d2c9a7703de24e4a8ab28f0f223dcbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:41:23.193490 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc0b5b691624439bb9b0951a21a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:23.193601 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc0b5b691624439bb9b0951a21a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:23.194000 12426 tablet_service.cc:1467] Processing CreateTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 (DEFAULT_TABLE table=test-table [id=d2c9a7703de24e4a8ab28f0f223dcbc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:41:23.195019 12426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc0b5b691624439bb9b0951a21a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:41:23.200196 12553 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: Bootstrap starting.
I20260806 02:41:23.201390 12554 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Bootstrap starting.
I20260806 02:41:23.201900 12555 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Bootstrap starting.
I20260806 02:41:23.202808 12553 tablet_bootstrap.cc:654] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:23.203819 12554 tablet_bootstrap.cc:654] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:23.204025 12555 tablet_bootstrap.cc:654] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Neither blocks nor log segments found. Creating new log.
I20260806 02:41:23.204154 12553 log.cc:824] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:23.204682 12554 log.cc:824] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:23.205289 12555 log.cc:824] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:41:23.206755 12553 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: No bootstrap required, opened a new log
I20260806 02:41:23.206950 12553 ts_tablet_manager.cc:1397] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:41:23.207219 12555 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: No bootstrap required, opened a new log
I20260806 02:41:23.207392 12555 ts_tablet_manager.cc:1397] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:41:23.210332 12554 tablet_bootstrap.cc:492] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: No bootstrap required, opened a new log
I20260806 02:41:23.210525 12554 ts_tablet_manager.cc:1397] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 02:41:23.210870 12553 raft_consensus.cc:348] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.210996 12555 raft_consensus.cc:348] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.211158 12553 raft_consensus.cc:374] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:23.211190 12555 raft_consensus.cc:374] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:23.211220 12553 raft_consensus.cc:729] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38aa71a69a1048c3b02612b9f27e61b6, State: Initialized, Role: FOLLOWER
I20260806 02:41:23.211234 12555 raft_consensus.cc:729] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe2d977ae8e44dc93fb3993d2c7717a, State: Initialized, Role: FOLLOWER
I20260806 02:41:23.211668 12555 consensus_queue.cc:260] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.211688 12553 consensus_queue.cc:260] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.212637 12555 ts_tablet_manager.cc:1428] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:41:23.212637 12553 ts_tablet_manager.cc:1428] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:41:23.213011 12527 heartbeater.cc:503] Master 127.28.191.62:41227 was elected leader, sending a full tablet report...
I20260806 02:41:23.213203 12124 heartbeater.cc:503] Master 127.28.191.62:41227 was elected leader, sending a full tablet report...
I20260806 02:41:23.213867 12554 raft_consensus.cc:348] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.214109 12554 raft_consensus.cc:374] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:41:23.214187 12554 raft_consensus.cc:729] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e5bd0a455a4f2e80c8028ca2b5fcf9, State: Initialized, Role: FOLLOWER
I20260806 02:41:23.214588 12554 consensus_queue.cc:260] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.215245 12322 heartbeater.cc:503] Master 127.28.191.62:41227 was elected leader, sending a full tablet report...
I20260806 02:41:23.215466 12554 ts_tablet_manager.cc:1428] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:41:23.238902 12566 raft_consensus.cc:482] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:41:23.239092 12566 raft_consensus.cc:504] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.240183 12566 leader_election.cc:288] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8e5bd0a455a4f2e80c8028ca2b5fcf9 (127.28.191.2:33817), 38aa71a69a1048c3b02612b9f27e61b6 (127.28.191.1:42359)
I20260806 02:41:23.244287 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.240395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:33817 (local address 127.28.191.3:42593)
0806 02:41:23.240495 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.240513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.240599 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.240976 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.240982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.241009 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:41:23.241201 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.241211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.242054 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.242057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.243006 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.243016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.243827 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.243831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.243938 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.243966 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.244062 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.244145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 02:41:23.244282 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.240576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42593 (local address 127.28.191.2:33817)
0806 02:41:23.240714 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.240718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.240733 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:41:23.240784 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.240787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.240836 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.240920 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:41:23.241314 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.241936 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.243148 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.243712 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.243804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.243843 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.244073 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.244160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:41:23.244555 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.240483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:41073 (local address 127.28.191.1:42359)
0806 02:41:23.240604 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.240609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.240989 (+ 380us) server_negotiation.cc:407] Connection header received
0806 02:41:23.241248 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.241254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.241311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.241380 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:41:23.241640 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.242250 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.243539 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.244037 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.244144 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.244186 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.244387 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.244465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:41:23.244930 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.240395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42359 (local address 127.28.191.3:41073)
0806 02:41:23.240903 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.240918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:41:23.241119 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:41:23.241396 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:41:23.241399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:41:23.241417 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:41:23.241529 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.241536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.242364 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.242366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.243369 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:41:23.243379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.244150 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.244153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:41:23.244250 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.244271 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:41:23.244701 (+ 430us) client_negotiation.cc:240] Negotiation successful
0806 02:41:23.244768 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":99,"thread_start_us":135,"threads_started":1}
I20260806 02:41:23.245247 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc0b5b691624439bb9b0951a21a99e9" candidate_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" is_pre_election: true
I20260806 02:41:23.245571 12253 raft_consensus.cc:2454] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe2d977ae8e44dc93fb3993d2c7717a in term 0.
I20260806 02:41:23.245667 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc0b5b691624439bb9b0951a21a99e9" candidate_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38aa71a69a1048c3b02612b9f27e61b6" is_pre_election: true
I20260806 02:41:23.245924 12042 raft_consensus.cc:2454] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe2d977ae8e44dc93fb3993d2c7717a in term 0.
I20260806 02:41:23.246035 12353 leader_election.cc:302] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe2d977ae8e44dc93fb3993d2c7717a, e8e5bd0a455a4f2e80c8028ca2b5fcf9; no voters:
I20260806 02:41:23.246316 12566 raft_consensus.cc:2781] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:41:23.246410 12566 raft_consensus.cc:482] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:41:23.246443 12566 raft_consensus.cc:3037] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:23.247685 12566 raft_consensus.cc:504] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.248064 12566 leader_election.cc:288] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [CANDIDATE]: Term 1 election: Requested vote from peers e8e5bd0a455a4f2e80c8028ca2b5fcf9 (127.28.191.2:33817), 38aa71a69a1048c3b02612b9f27e61b6 (127.28.191.1:42359)
I20260806 02:41:23.248400 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc0b5b691624439bb9b0951a21a99e9" candidate_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9"
I20260806 02:41:23.248466 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc0b5b691624439bb9b0951a21a99e9" candidate_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38aa71a69a1048c3b02612b9f27e61b6"
I20260806 02:41:23.248512 12253 raft_consensus.cc:3037] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:23.248621 12042 raft_consensus.cc:3037] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:41:23.249667 12253 raft_consensus.cc:2454] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe2d977ae8e44dc93fb3993d2c7717a in term 1.
I20260806 02:41:23.249769 12042 raft_consensus.cc:2454] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe2d977ae8e44dc93fb3993d2c7717a in term 1.
I20260806 02:41:23.249938 12353 leader_election.cc:302] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe2d977ae8e44dc93fb3993d2c7717a, e8e5bd0a455a4f2e80c8028ca2b5fcf9; no voters:
I20260806 02:41:23.250176 12566 raft_consensus.cc:2781] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:41:23.250351 12566 raft_consensus.cc:686] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 1 LEADER]: Becoming Leader. State: Replica: bfe2d977ae8e44dc93fb3993d2c7717a, State: Running, Role: LEADER
I20260806 02:41:23.250660 12566 consensus_queue.cc:237] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } }
I20260806 02:41:23.253772 11834 catalog_manager.cc:5495] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe2d977ae8e44dc93fb3993d2c7717a (127.28.191.3). New cstate: current_term: 1 leader_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe2d977ae8e44dc93fb3993d2c7717a" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 } health_report { overall_health: UNKNOWN } } }
I20260806 02:41:23.270579 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.266903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.28.191.3:36873)
0806 02:41:23.267048 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:41:23.267054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:41:23.267085 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:41:23.267271 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:41:23.267274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:41:23.267327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:41:23.267429 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:41:23.267695 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.268194 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.269110 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:41:23.269328 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:41:23.269443 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:41:23.269769 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:41:23.270174 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:41:23.270307 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:41:23.270366 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 02:41:23.270420 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:41:23.276088 12042 raft_consensus.cc:1261] T fbc0b5b691624439bb9b0951a21a99e9 P 38aa71a69a1048c3b02612b9f27e61b6 [term 1 FOLLOWER]: Refusing update from remote peer bfe2d977ae8e44dc93fb3993d2c7717a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:41:23.276129 12253 raft_consensus.cc:1261] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 1 FOLLOWER]: Refusing update from remote peer bfe2d977ae8e44dc93fb3993d2c7717a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:41:23.276868 12566 consensus_queue.cc:1035] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38aa71a69a1048c3b02612b9f27e61b6" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:41:23.277120 12571 consensus_queue.cc:1035] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e5bd0a455a4f2e80c8028ca2b5fcf9" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:41:23.287762 12577 mvcc.cc:205] Tried to move back new op lower bound from 7315390805092024320 to 7315390804997046272. Current Snapshot: MvccSnapshot[applied={T|T < 7315390805092024320}]
I20260806 02:41:23.292917 11837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:41:23.293116 11837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:41:23.296463 11837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e2f8ec17ccb4ba5be97e965fd432195: Sending DeleteTablet for 3 replicas of tablet fbc0b5b691624439bb9b0951a21a99e9
I20260806 02:41:23.297300 12426 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:41:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:50772
I20260806 02:41:23.297742 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:41:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:58330
I20260806 02:41:23.298527 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11926
I20260806 02:41:23.298463 12225 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc0b5b691624439bb9b0951a21a99e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:41:23 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57280
I20260806 02:41:23.309319 12590 tablet_replica.cc:330] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: stopping tablet replica
I20260806 02:41:23.309702 12590 raft_consensus.cc:2229] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:41:23.310117 12590 raft_consensus.cc:2258] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:41:23.311730 12590 ts_tablet_manager.cc:1905] T fbc0b5b691624439bb9b0951a21a99e9 P e8e5bd0a455a4f2e80c8028ca2b5fcf9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:41:23.313740 11822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:42359 (error 108)
I20260806 02:41:23.314658 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 12130
I20260806 02:41:23.316347 12587 tablet_replica.cc:330] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: stopping tablet replica
I20260806 02:41:23.316802 12587 raft_consensus.cc:2229] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:41:23.317329 12587 raft_consensus.cc:2258] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:41:23.321684 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.321354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42359 (local address 127.0.0.1:58338)
0806 02:41:23.321499 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.321601 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:41:23.321853 11822 catalog_manager.cc:4584] TS 38aa71a69a1048c3b02612b9f27e61b6 (127.28.191.1:42359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbc0b5b691624439bb9b0951a21a99e9: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42359: connect: Connection refused (error 111)
I20260806 02:41:23.323041 12587 ts_tablet_manager.cc:1905] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:41:23.326647 12587 ts_tablet_manager.cc:1918] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:41:23.326771 12587 log.cc:1196] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/fbc0b5b691624439bb9b0951a21a99e9
I20260806 02:41:23.327142 12587 ts_tablet_manager.cc:1939] T fbc0b5b691624439bb9b0951a21a99e9 P bfe2d977ae8e44dc93fb3993d2c7717a: Deleting consensus metadata
I20260806 02:41:23.327602 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:41:23.327348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:33817 (local address 127.0.0.1:57296)
0806 02:41:23.327466 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:41:23.327522 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:41:23.327880 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 12327
I20260806 02:41:23.340870 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 11796
2026-08-06T02:41:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2746 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:41:23 dist-test-slave-tfpf krb5kdc[12598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:41:23 dist-test-slave-tfpf krb5kdc[12598](info): set up 2 sockets
Aug 06 02:41:23 dist-test-slave-tfpf krb5kdc[12598](info): commencing operation
krb5kdc: starting...
W20260806 02:41:24.550684 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:41:24 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:41:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangeradmin_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangeradmin_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangerlookup_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangerlookup_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
I20260806 02:41:24.715099 29436 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklwD5TR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/postgres -l logfile start
2026-08-06 02:41:26.951 UTC [12636] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:41:26.951 UTC [12636] LOG: listening on IPv4 address "127.28.191.20", port 32779
2026-08-06 02:41:26.955 UTC [12636] LOG: listening on Unix socket "/tmp/.s.PGSQL.32779"
2026-08-06 02:41:26.980 UTC [12643] LOG: database system was shut down at 2026-08-06 02:41:25 UTC
2026-08-06 02:41:26.984 UTC [12636] LOG: database system is ready to accept connections
I20260806 02:41:28.090456 29436 mini_postgres.cc:96] Postgres bound to 32779
2026-08-06 02:41:28.098 UTC [12657] FATAL: database "slave" does not exist
127.28.191.20:32779 - accepting connections
I20260806 02:41:28.098927 29436 mini_ranger.cc:161] Starting Ranger...
I20260806 02:41:28.114203 29436 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:41:28.529665 29436 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:41:28.529757 29436 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin
2026-08-06 02:41:28,813 [I] DB FLAVOR :POSTGRES
2026-08-06 02:41:28,814 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:41:28,814 [I] Checking connection..
2026-08-06 02:41:28,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:41:29,048 [I] Checking connection passed.
2026-08-06 02:41:29,048 [I] --------- Verifying version history table ---------
2026-08-06 02:41:29,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:41:29,297 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:41:29,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:41:29,546 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:41:29,546 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:41:29,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:41:29.770 UTC [12748] WARNING: there is no transaction in progress
2026-08-06 02:41:29,777 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:41:29,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:41:30,040 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:41:30,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:41:30,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:41:30,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:41:30,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tfpf.c.gcp-upstream.internal','N') ;"
2026-08-06 02:41:30,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:41:31,040 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:41:31,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:41:31,284 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:41:31,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:41:31,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:41:31,784 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:41:31,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:41:32.700 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.716 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.731 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.745 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.760 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.777 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.830 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.837 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.850 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.860 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.870 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.882 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.891 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.897 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.903 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:32.909 UTC [12909] WARNING: there is no transaction in progress
2026-08-06 02:41:33,493 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:41:33,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:41:33,753 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:41:33,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:41:34,023 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:41:34,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:41:34,268 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:41:34,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:41:34,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:41:34,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tfpf.c.gcp-upstream.internal';"
2026-08-06 02:41:35,022 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:41:35,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:41:35,281 [I] DB_PATCHES have already been applied
I20260806 02:41:35.287408 29436 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklwD5TR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:41:35.287516 29436 mini_ranger.cc:193] Using host: 127.28.191.20
I20260806 02:41:35.292060 29436 mini_ranger.cc:227] Ranger admin URL: http://127.28.191.20:58453
Aug 06, 2026 2:41:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:41:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:41:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:41:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:41:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:41:36.625249 29436 mini_ranger.cc:160] Time spent starting Ranger: real 8.526s user 0.002s sys 0.011s
Aug 06, 2026 2:41:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:41:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 2:41:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:41:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:41:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:41:57.035657 29436 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangerkms_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/rangerkms_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.191.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.191.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:41:57 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:41:57.141785 29436 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:41:57.152678 29436 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:41:57.878355 29436 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:41:57.878468 29436 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms
I20260806 02:41:57.878585 29436 mini_ranger_kms.cc:227] Using postgres at 127.28.191.20:32779
2026-08-06 02:41:58,157 [I] DB FLAVOR :POSTGRES
2026-08-06 02:41:58,157 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:41:58,157 [I] Checking connection
2026-08-06 02:41:58,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:41:58,408 [I] connection success
2026-08-06 02:41:58,409 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:41:58,409 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:41:58,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:41:58,644 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:41:58,644 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:41:58,644 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:41:58,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:32779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:41:58,890 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:41:59.349624 29436 mini_ranger_kms.cc:310] Created kms service
I20260806 02:41:59.516743 29436 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:41:59.557175 29436 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:41:59 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:41:59.936908 29436 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:41:59.937104 29436 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:41:59.937132 29436 mini_ranger_kms.cc:240] Using host: 127.28.191.20
I20260806 02:41:59.942163 29436 mini_ranger_kms.cc:276] Ranger KMS PID: 13202
I20260806 02:41:59.942322 29436 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.191.20:42793
Aug 06, 2026 2:42:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:42:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:42:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:42:02 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:02.614812 29436 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.473s user 0.002s sys 0.260s
I20260806 02:42:02.614945 29436 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:42:02.615056 29436 mini_ranger_kms.cc:401] 127.28.191.20:42793/kms/v1/keys
Aug 06 02:42:02 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
Aug 06 02:42:03 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
Aug 06 02:42:05 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:42:05.516552 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36501
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36501
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:42793/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:05.631367 13285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:05.631639 13285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:05.631681 13285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:05.635680 13285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:42:05.635735 13285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:05.635753 13285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:05.635769 13285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:42:05.635819 13285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:42:05.641187 13285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.191.20:42793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36501
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:05.642184 13285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:05.643187 13285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:05.652757 13292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:05.652779 13297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:05.652827 13293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:05.653182 13285 server_base.cc:1034] running on GCE node
I20260806 02:42:05.654712 13285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:05.655999 13285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:05.657138 13285 hybrid_clock.cc:648] HybridClock initialized: now 1785984125657091 us; error 53 us; skew 500 ppm
Aug 06 02:42:05 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:05.660447 13285 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:42:05.661386 13285 webserver.cc:460] Webserver started at http://127.28.191.62:43373/ using document root <none> and password file <none>
I20260806 02:42:05.661911 13285 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:05.661966 13285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:05.662185 13285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:05 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984125, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:05.708349 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "fe347837852d4eeb8ee882f2c8abdf03"
format_stamp: "Formatted at 2026-08-06 02:42:05 on dist-test-slave-tfpf"
server_key: "43e43a8089c1e28bd2ca2f76cca19dfa"
server_key_iv: "88c15c6ad531b00305ed718df5813f7f"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:05.708981 13285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "fe347837852d4eeb8ee882f2c8abdf03"
format_stamp: "Formatted at 2026-08-06 02:42:05 on dist-test-slave-tfpf"
server_key: "43e43a8089c1e28bd2ca2f76cca19dfa"
server_key_iv: "88c15c6ad531b00305ed718df5813f7f"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:05.713181 13285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:42:05.726282 13307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:05.727661 13285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:42:05.727793 13285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "fe347837852d4eeb8ee882f2c8abdf03"
format_stamp: "Formatted at 2026-08-06 02:42:05 on dist-test-slave-tfpf"
server_key: "43e43a8089c1e28bd2ca2f76cca19dfa"
server_key_iv: "88c15c6ad531b00305ed718df5813f7f"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:05.727919 13285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:05.740379 13285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:05.743188 13285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:05.743376 13285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:05.751356 13285 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36501
I20260806 02:42:05.751461 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36501 every 8 connection(s)
I20260806 02:42:05.752483 13285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:42:05.753752 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 13285
I20260806 02:42:05.753873 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:42:05.754168 29436 external_mini_cluster.cc:1428] Setting key 69ce10aaa3ebc8a1f8e0055ce68bb7d0
I20260806 02:42:05.755909 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:42:05 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:05.763180 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: Bootstrap starting.
I20260806 02:42:05.765899 13384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:05.767194 13384 log.cc:824] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:05.767653 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:05.755483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.28.191.62:36501)
0806 02:42:05.755849 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:05.755879 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:05.755911 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:42:05.756592 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:05.756607 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:05.756932 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:05.757291 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:05.758114 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:05.759054 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:05.759969 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:05.760434 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:05.760610 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:05.763141 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:05.763146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:05.763156 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:05.763180 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:05.764826 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:05.765359 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:05.765364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:05.765368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:05.765462 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:05.765732 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:05.765736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:05.765738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:05.766179 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:05.766376 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:05.766674 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 02:42:05.766937 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
I20260806 02:42:05.769402 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: No bootstrap required, opened a new log
I20260806 02:42:05.773455 13384 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } }
I20260806 02:42:05.773725 13384 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:05.773784 13384 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe347837852d4eeb8ee882f2c8abdf03, State: Initialized, Role: FOLLOWER
I20260806 02:42:05.774271 13384 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } }
I20260806 02:42:05.774425 13384 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:42:05.774509 13384 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:42:05.774611 13384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:05.775633 13384 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } }
I20260806 02:42:05.776000 13384 leader_election.cc:302] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe347837852d4eeb8ee882f2c8abdf03; no voters:
I20260806 02:42:05.776331 13384 leader_election.cc:288] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:42:05.776438 13390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:42:05.776712 13390 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [term 1 LEADER]: Becoming Leader. State: Replica: fe347837852d4eeb8ee882f2c8abdf03, State: Running, Role: LEADER
I20260806 02:42:05.777097 13390 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } }
I20260806 02:42:05.777621 13384 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:42:05.778761 13396 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe347837852d4eeb8ee882f2c8abdf03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } } }
I20260806 02:42:05.778895 13396 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:05.779294 13404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:42:05.779260 13395 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe347837852d4eeb8ee882f2c8abdf03. Latest consensus state: current_term: 1 leader_uuid: "fe347837852d4eeb8ee882f2c8abdf03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe347837852d4eeb8ee882f2c8abdf03" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36501 } } }
I20260806 02:42:05.779448 13395 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03 [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:05.782629 13404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:42:05.783181 13384 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:42:05.792097 13404 catalog_manager.cc:1347] Generated new cluster ID: a5b35e23283d4d019961b56f8d6d6481
I20260806 02:42:05.792317 13404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:42:05.807125 13404 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:42:05.808389 13404 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:42:05.818513 13404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: Generated new TSK 0
I20260806 02:42:05.819409 13404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:42:06.638604 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:42793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:06.792536 13453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:06.792760 13453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:06.792799 13453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:06.797063 13453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:06.797212 13453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:06.797315 13453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:42:06.805254 13453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.191.20:42793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:06.806653 13453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:06.807974 13453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:06.819082 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:06.819145 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:06.819496 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:06.819722 13453 server_base.cc:1034] running on GCE node
I20260806 02:42:06.820294 13453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:06.821096 13453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:06.822297 13453 hybrid_clock.cc:648] HybridClock initialized: now 1785984126822272 us; error 40 us; skew 500 ppm
Aug 06 02:42:06 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984126, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:06.825824 13453 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:42:06.826967 13453 webserver.cc:460] Webserver started at http://127.28.191.1:40943/ using document root <none> and password file <none>
I20260806 02:42:06.827751 13453 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:06.827855 13453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:06.828123 13453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:06 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984126, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:06.844683 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "0970e30c47d040f789d91d08ef37fcd5"
format_stamp: "Formatted at 2026-08-06 02:42:06 on dist-test-slave-tfpf"
server_key: "976fe488d7a0b9f179344eb02ac25e63"
server_key_iv: "6080964e81cc24c6c494cb07f3b01ab0"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:06.845767 13453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "0970e30c47d040f789d91d08ef37fcd5"
format_stamp: "Formatted at 2026-08-06 02:42:06 on dist-test-slave-tfpf"
server_key: "976fe488d7a0b9f179344eb02ac25e63"
server_key_iv: "6080964e81cc24c6c494cb07f3b01ab0"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:06.850076 13453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:42:06.872716 13478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:06.874358 13453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:42:06.874490 13453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "0970e30c47d040f789d91d08ef37fcd5"
format_stamp: "Formatted at 2026-08-06 02:42:06 on dist-test-slave-tfpf"
server_key: "976fe488d7a0b9f179344eb02ac25e63"
server_key_iv: "6080964e81cc24c6c494cb07f3b01ab0"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:06.874636 13453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:06.902325 13453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:06.912559 13453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:06.912832 13453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:06.913662 13453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:06.914772 13453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:06.914881 13453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:06.915015 13453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:06.915104 13453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:06.926775 13453 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:44177
I20260806 02:42:06.927922 13453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:42:06.929831 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:44177 every 8 connection(s)
I20260806 02:42:06.929865 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 13453
I20260806 02:42:06.929982 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:42:06.930343 29436 external_mini_cluster.cc:1428] Setting key bd45cea2fd8a93db531e649a00e87449
Aug 06 02:42:06 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984126, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:06.952694 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:06.933845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:53991 (local address 127.28.191.62:36501)
0806 02:42:06.935796 (+ 1951us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:06.935800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:06.935813 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:42:06.935861 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:06.935864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:06.935905 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:06.935980 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:06.937499 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:06.938364 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:06.940052 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:06.940440 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:06.940544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:06.947696 (+ 7152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:06.947714 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:06.947716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:06.947746 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:06.950031 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:06.950812 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:06.950816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:06.950817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:06.950909 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:06.951311 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:06.951314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:06.951316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:06.951564 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:06.951653 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:06.952343 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 02:42:06.952489 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:42:06.954330 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:06.932654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36501 (local address 127.28.191.1:53991)
0806 02:42:06.934356 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:06.934412 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:06.935387 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:06.936270 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:06.936279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:06.936606 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:06.937334 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:06.937354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:06.938509 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:06.938513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:06.939893 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:06.939905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:06.940726 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:06.940730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:06.940929 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:06.941657 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:06.941678 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:06.947482 (+ 5804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:06.950197 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:06.950206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:06.950229 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:06.950666 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:06.951012 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:06.951015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:06.951017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:06.951199 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:06.951708 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:06.951715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:06.952037 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 02:42:06.952873 (+ 836us) client_negotiation.cc:240] Negotiation successful
0806 02:42:06.953275 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":143,"threads_started":1}
I20260806 02:42:06.955899 13658 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36501
I20260806 02:42:06.956179 13658 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:06.956779 13658 heartbeater.cc:511] Master 127.28.191.62:36501 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:42:06.958556 13336 ts_manager.cc:194] Registered new tserver with Master: 0970e30c47d040f789d91d08ef37fcd5 (127.28.191.1:44177)
I20260806 02:42:06.959709 13336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:53991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:42:06.985663 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:42793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:42:07.160765 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:07.161223 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:07.161352 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:07.167264 13667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:07.167387 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:07.167523 13667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:42:07.175149 13667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.191.20:42793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:07.176694 13667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:07.178226 13667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:07.191342 13679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:07.191965 13681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:07.192480 13667 server_base.cc:1034] running on GCE node
W20260806 02:42:07.193161 13678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:07.193715 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:07.194504 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:07.202387 13667 hybrid_clock.cc:648] HybridClock initialized: now 1785984127202303 us; error 94 us; skew 500 ppm
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:07.206158 13667 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:42:07.207422 13667 webserver.cc:460] Webserver started at http://127.28.191.2:38971/ using document root <none> and password file <none>
I20260806 02:42:07.208186 13667 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:07.208326 13667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:07.208603 13667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:07.228475 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "3d3c5f69dea147389e04e794677a9d21"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "a9ae915622000dc16195c8b2818feb06"
server_key_iv: "2025495bbd8a57e1102d63cc3720eb18"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.230962 13667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "3d3c5f69dea147389e04e794677a9d21"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "a9ae915622000dc16195c8b2818feb06"
server_key_iv: "2025495bbd8a57e1102d63cc3720eb18"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.236464 13667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:42:07.257660 13690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.259433 13667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:42:07.259636 13667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "3d3c5f69dea147389e04e794677a9d21"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "a9ae915622000dc16195c8b2818feb06"
server_key_iv: "2025495bbd8a57e1102d63cc3720eb18"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.259889 13667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:07.308075 13667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:07.314388 13667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:07.314661 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:07.315439 13667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:07.316552 13667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:07.316922 13667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.317042 13667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:07.317134 13667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.328567 13667 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:42813
I20260806 02:42:07.329239 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:42813 every 8 connection(s)
I20260806 02:42:07.329739 13667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:42:07.330686 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 13667
I20260806 02:42:07.330806 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:42:07.331130 29436 external_mini_cluster.cc:1428] Setting key 8384bb7c082a27eb4bbfe298aba5c12c
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:42:07.351989 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.334736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:49115 (local address 127.28.191.62:36501)
0806 02:42:07.334899 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.334904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.335241 (+ 337us) server_negotiation.cc:407] Connection header received
0806 02:42:07.336361 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.336365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.336415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.336494 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:07.338723 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.339509 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.341264 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.341626 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.341739 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.347118 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:07.347123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:07.347127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:07.347163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:07.349241 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.349914 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.349920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.349922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.350026 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.350523 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.350527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.350529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.350808 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:07.350920 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.351578 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.351744 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:42:07.352665 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.333650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36501 (local address 127.28.191.2:49115)
0806 02:42:07.335084 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.335139 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.336142 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.337503 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.337513 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.337896 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:07.338550 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.338570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.339653 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.339657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.341123 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.341136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.341810 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.341814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.342019 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.344867 (+ 2848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:07.344890 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:07.346884 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:07.349390 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:07.349400 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:07.349420 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:07.349772 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:07.350135 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:07.350141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:07.350143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:07.350378 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:07.350928 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:07.350933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:07.351280 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.351446 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.351712 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
I20260806 02:42:07.353993 13872 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36501
I20260806 02:42:07.354276 13872 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:07.354779 13872 heartbeater.cc:511] Master 127.28.191.62:36501 requested a full tablet report, sending...
I20260806 02:42:07.355962 13336 ts_manager.cc:194] Registered new tserver with Master: 3d3c5f69dea147389e04e794677a9d21 (127.28.191.2:42813)
I20260806 02:42:07.356786 13336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:49115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:42:07.376089 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:42793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:07.521199 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:07.521641 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:07.521777 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
W20260806 02:42:07.528376 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:07.528559 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:07.528702 13879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:42:07.535917 13879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.191.20:42793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:07.537586 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:07.538972 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:07.555281 13889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:07.559134 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:07.561246 13888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:07.563654 13879 server_base.cc:1034] running on GCE node
I20260806 02:42:07.564324 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:07.565251 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:07.566591 13879 hybrid_clock.cc:648] HybridClock initialized: now 1785984127566526 us; error 77 us; skew 500 ppm
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:07.571317 13879 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:42:07.573289 13879 webserver.cc:460] Webserver started at http://127.28.191.3:43765/ using document root <none> and password file <none>
I20260806 02:42:07.574038 13879 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:07.574152 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:07.574401 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:07.595312 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "fc0c40687ce6428675c013db126262a3"
server_key_iv: "542b21f7f8412746dd2ac6f08c0facda"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.596378 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "fc0c40687ce6428675c013db126262a3"
server_key_iv: "542b21f7f8412746dd2ac6f08c0facda"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.602420 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:42:07.635344 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.637199 13879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:42:07.637367 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9"
format_stamp: "Formatted at 2026-08-06 02:42:07 on dist-test-slave-tfpf"
server_key: "fc0c40687ce6428675c013db126262a3"
server_key_iv: "542b21f7f8412746dd2ac6f08c0facda"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:07.637522 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:07.672245 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:07.675745 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:07.676035 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:07.676880 13879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:07.678087 13879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:07.678200 13879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.678305 13879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:07.678376 13879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:07.694038 13879 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36229
I20260806 02:42:07.695322 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:42:07.702404 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 13879
I20260806 02:42:07.702550 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:42:07.702895 29436 external_mini_cluster.cc:1428] Setting key d6266a4256cc68ac5fea39f138484889
I20260806 02:42:07.697094 14081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36229 every 8 connection(s)
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:07.716279 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.700856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:45495 (local address 127.28.191.62:36501)
0806 02:42:07.701012 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.701018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.701665 (+ 647us) server_negotiation.cc:407] Connection header received
0806 02:42:07.703045 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.703050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.703113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.703177 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:07.704945 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.705810 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.707797 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.708148 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.708239 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.711514 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:07.711519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:07.711522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:07.711554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:07.713515 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.714234 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.714238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.714241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.714339 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.714838 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.714842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.714844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.715106 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:07.715194 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.715926 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.716071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:42:07.717983 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.699694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36501 (local address 127.28.191.3:45495)
0806 02:42:07.701482 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.701540 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.702753 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.703672 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.703681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.704064 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:07.704760 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.704782 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.706232 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.706247 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.707635 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.707650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.708457 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.708462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.708674 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.709413 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:07.709436 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:07.711293 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:07.713669 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:07.713676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:07.713692 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:07.714091 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:07.714481 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:07.714484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:07.714486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:07.714699 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:07.715300 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:07.715307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:07.715620 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.716500 (+ 880us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.716869 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"thread_start_us":164,"threads_started":1}
I20260806 02:42:07.719287 14082 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36501
I20260806 02:42:07.719537 14082 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:07.720136 14082 heartbeater.cc:511] Master 127.28.191.62:36501 requested a full tablet report, sending...
I20260806 02:42:07.721355 13336 ts_manager.cc:194] Registered new tserver with Master: 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 (127.28.191.3:36229)
I20260806 02:42:07.721927 13336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:45495
I20260806 02:42:07.729393 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:42:07.747839 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.733565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.28.191.62:36501)
0806 02:42:07.733688 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.733693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.737397 (+ 3704us) server_negotiation.cc:407] Connection header received
0806 02:42:07.737630 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.737634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.737689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.737752 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:07.738957 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.739767 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.741160 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.741504 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.741615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.743774 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:07.743779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:07.743782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:07.743811 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:07.745661 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.746275 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.746279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.746282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.746381 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:07.746747 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:07.746751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:07.746753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:07.747002 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:07.747123 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.747532 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.747656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:42:07.753995 13336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:42:07.759999 13336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:42:07.785214 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.774588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:44177 (local address 127.0.0.1:59820)
0806 02:42:07.779181 (+ 4593us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.779236 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.779370 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.780305 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.780312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.780337 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:07.780465 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.780472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.781948 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.781952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.783332 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.783345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.784632 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.784636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.784757 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.784795 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.784905 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.785022 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3452,"thread_start_us":129,"threads_started":1}
I20260806 02:42:07.786149 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.775346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.28.191.1:44177)
0806 02:42:07.779659 (+ 4313us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.779667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.779691 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:42:07.779771 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.779779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.779991 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.780132 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:07.780599 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.781812 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.783479 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.784316 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.784420 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.784569 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.785883 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.785990 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4195,"thread_start_us":154,"threads_started":1}
I20260806 02:42:07.789549 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 609bc80c925c463e839df5313eb06e6e (DEFAULT_TABLE table=test-table [id=0911a18d285040e7b3e7f07a5960ef86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:07.790751 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bc80c925c463e839df5313eb06e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:07.792932 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.778459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42813 (local address 127.0.0.1:38500)
0806 02:42:07.785217 (+ 6758us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.785233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.785333 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.786302 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.786305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.786322 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:07.786412 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.786421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.790054 (+ 3633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.790057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.791193 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.791204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.792525 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.792529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.792612 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.792635 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.792708 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.792775 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6698,"thread_start_us":108,"threads_started":1}
I20260806 02:42:07.800745 14109 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Bootstrap starting.
I20260806 02:42:07.803283 14109 tablet_bootstrap.cc:654] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:07.804270 14109 log.cc:824] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:07.805694 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.777114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:36229 (local address 127.0.0.1:39010)
0806 02:42:07.783631 (+ 6517us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.783648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.783751 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.787160 (+ 3409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.787164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.787183 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:07.787275 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.787281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.793235 (+ 5954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.793238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.794364 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.794376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.805210 (+ 10834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.805215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.805359 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.805392 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.805456 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.805543 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6444,"thread_start_us":102,"threads_started":1}
I20260806 02:42:07.806608 14109 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: No bootstrap required, opened a new log
I20260806 02:42:07.806761 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.777609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.28.191.3:36229)
0806 02:42:07.786711 (+ 9102us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.786723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.786745 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:42:07.786814 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.786819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.786985 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.787433 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:07.787573 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.788634 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.794574 (+ 5940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.795404 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.806229 (+ 10825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.806416 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.806478 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.806592 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8988,"thread_start_us":116,"threads_started":1}
I20260806 02:42:07.806828 14109 ts_tablet_manager.cc:1397] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:42:07.807209 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.778923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.28.191.2:42813)
0806 02:42:07.785592 (+ 6669us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.785612 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.785639 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:42:07.785721 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.785729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.785961 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.786116 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:07.788785 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.789912 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.791373 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.792155 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.792276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.792455 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.806861 (+ 14406us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.807017 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6551,"thread_start_us":111,"threads_started":1}
I20260806 02:42:07.809945 13977 tablet_service.cc:1467] Processing CreateTablet for tablet 609bc80c925c463e839df5313eb06e6e (DEFAULT_TABLE table=test-table [id=0911a18d285040e7b3e7f07a5960ef86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:07.810176 13768 tablet_service.cc:1467] Processing CreateTablet for tablet 609bc80c925c463e839df5313eb06e6e (DEFAULT_TABLE table=test-table [id=0911a18d285040e7b3e7f07a5960ef86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:07.811082 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bc80c925c463e839df5313eb06e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:07.811261 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609bc80c925c463e839df5313eb06e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:07.811396 14109 raft_consensus.cc:348] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.811674 14109 raft_consensus.cc:374] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:07.811726 14109 raft_consensus.cc:729] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0970e30c47d040f789d91d08ef37fcd5, State: Initialized, Role: FOLLOWER
I20260806 02:42:07.812199 14109 consensus_queue.cc:260] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.813251 14109 ts_tablet_manager.cc:1428] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:42:07.813589 13658 heartbeater.cc:503] Master 127.28.191.62:36501 was elected leader, sending a full tablet report...
I20260806 02:42:07.821216 14113 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Bootstrap starting.
I20260806 02:42:07.824165 14113 tablet_bootstrap.cc:654] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:07.825587 14113 log.cc:824] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:07.829453 14114 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Bootstrap starting.
I20260806 02:42:07.832484 14114 tablet_bootstrap.cc:654] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:07.833395 14114 log.cc:824] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:07.835768 14114 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: No bootstrap required, opened a new log
I20260806 02:42:07.835965 14114 ts_tablet_manager.cc:1397] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:42:07.839907 14114 raft_consensus.cc:348] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.840174 14114 raft_consensus.cc:374] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:07.840221 14114 raft_consensus.cc:729] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9, State: Initialized, Role: FOLLOWER
I20260806 02:42:07.840857 14114 consensus_queue.cc:260] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.842074 14114 ts_tablet_manager.cc:1428] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 02:42:07.842499 14082 heartbeater.cc:503] Master 127.28.191.62:36501 was elected leader, sending a full tablet report...
I20260806 02:42:07.844437 14113 tablet_bootstrap.cc:492] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: No bootstrap required, opened a new log
I20260806 02:42:07.844677 14113 ts_tablet_manager.cc:1397] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260806 02:42:07.848811 14113 raft_consensus.cc:348] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.849196 14113 raft_consensus.cc:374] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:07.849309 14113 raft_consensus.cc:729] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3c5f69dea147389e04e794677a9d21, State: Initialized, Role: FOLLOWER
I20260806 02:42:07.849892 14113 consensus_queue.cc:260] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.851047 14113 ts_tablet_manager.cc:1428] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:42:07.851459 13872 heartbeater.cc:503] Master 127.28.191.62:36501 was elected leader, sending a full tablet report...
I20260806 02:42:07.880811 14119 raft_consensus.cc:482] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:42:07.881078 14119 raft_consensus.cc:504] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.882375 14119 leader_election.cc:288] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0970e30c47d040f789d91d08ef37fcd5 (127.28.191.1:44177), 3d3c5f69dea147389e04e794677a9d21 (127.28.191.2:42813)
I20260806 02:42:07.888159 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.882767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:44177 (local address 127.28.191.3:57511)
0806 02:42:07.883448 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.883462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.883546 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.884391 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.884394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.884414 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:07.884524 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.884530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.885574 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.885577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.886788 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.886799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.887767 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.887770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.887867 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.887890 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.887959 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.888027 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604}
I20260806 02:42:07.889323 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.883981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:57511 (local address 127.28.191.1:44177)
0806 02:42:07.884089 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.884093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.884105 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:42:07.884156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.884159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.884203 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.884271 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:07.884650 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.885440 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.886928 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.887529 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.887639 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.887707 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.889093 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.889190 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:42:07.890726 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bc80c925c463e839df5313eb06e6e" candidate_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0970e30c47d040f789d91d08ef37fcd5" is_pre_election: true
I20260806 02:42:07.891054 13594 raft_consensus.cc:2454] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 in term 0.
I20260806 02:42:07.891698 13904 leader_election.cc:302] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0970e30c47d040f789d91d08ef37fcd5, 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9; no voters:
I20260806 02:42:07.892056 14119 raft_consensus.cc:2781] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:42:07.892138 14119 raft_consensus.cc:482] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:42:07.892175 14119 raft_consensus.cc:3037] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:07.893548 14119 raft_consensus.cc:504] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.894260 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bc80c925c463e839df5313eb06e6e" candidate_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0970e30c47d040f789d91d08ef37fcd5"
I20260806 02:42:07.894371 13594 raft_consensus.cc:3037] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:07.895629 13594 raft_consensus.cc:2454] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 in term 1.
I20260806 02:42:07.896077 13904 leader_election.cc:302] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0970e30c47d040f789d91d08ef37fcd5, 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9; no voters:
I20260806 02:42:07.898676 14119 leader_election.cc:288] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0970e30c47d040f789d91d08ef37fcd5 (127.28.191.1:44177), 3d3c5f69dea147389e04e794677a9d21 (127.28.191.2:42813)
I20260806 02:42:07.898849 14119 raft_consensus.cc:2781] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:42:07.898957 14119 raft_consensus.cc:686] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9, State: Running, Role: LEADER
I20260806 02:42:07.899365 14119 consensus_queue.cc:237] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } }
I20260806 02:42:07.900272 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.883777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60333 (local address 127.28.191.2:42813)
0806 02:42:07.883886 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.883890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.888559 (+ 4669us) server_negotiation.cc:407] Connection header received
0806 02:42:07.888616 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.888619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.888668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.888749 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:07.889081 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.889852 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.897742 (+ 7890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.898316 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.898429 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.898480 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.900074 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.900160 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:42:07.899996 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.882998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42813 (local address 127.28.191.3:60333)
0806 02:42:07.888306 (+ 5308us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:07.888320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:07.888394 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:07.888855 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:07.888857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:07.888873 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:07.888939 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.888944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.896417 (+ 7473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.896420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.897575 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:07.897587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.899575 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.899579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:07.899668 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.899707 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:42:07.899779 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:42:07.899844 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5256,"thread_start_us":108,"threads_started":1}
I20260806 02:42:07.902034 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bc80c925c463e839df5313eb06e6e" candidate_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3c5f69dea147389e04e794677a9d21" is_pre_election: true
I20260806 02:42:07.902315 13803 raft_consensus.cc:2454] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 in term 0.
I20260806 02:42:07.902612 13802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609bc80c925c463e839df5313eb06e6e" candidate_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3c5f69dea147389e04e794677a9d21"
I20260806 02:42:07.902715 13802 raft_consensus.cc:3037] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:07.903941 13802 raft_consensus.cc:2454] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 in term 1.
I20260806 02:42:07.908511 13336 catalog_manager.cc:5495] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fd3cd1d5e2f4bee8d3e3dc70f4f68a9" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:42:07 dist-test-slave-tfpf krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984127, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
W20260806 02:42:07.931504 13659 tablet.cc:2369] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:42:07.944006 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:07.939621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.28.191.3:36229)
0806 02:42:07.939755 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:07.939759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:07.939863 (+ 104us) server_negotiation.cc:407] Connection header received
0806 02:42:07.940019 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:07.940023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:07.940093 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:07.940203 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:42:07.940482 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.941117 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.942287 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:07.942599 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:07.942734 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:07.943112 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:42:07.943530 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:42:07.943649 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:07.943770 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:42:07.943841 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260806 02:42:07.947742 14083 tablet.cc:2369] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:42:07.951527 13594 raft_consensus.cc:1261] T 609bc80c925c463e839df5313eb06e6e P 0970e30c47d040f789d91d08ef37fcd5 [term 1 FOLLOWER]: Refusing update from remote peer 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:42:07.951527 13802 raft_consensus.cc:1261] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 1 FOLLOWER]: Refusing update from remote peer 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:42:07.952550 14119 consensus_queue.cc:1035] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3c5f69dea147389e04e794677a9d21" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:42:07.952773 14127 consensus_queue.cc:1035] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0970e30c47d040f789d91d08ef37fcd5" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 44177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:42:07.967054 14130 mvcc.cc:205] Tried to move back new op lower bound from 7315390988080005120 to 7315390987902877696. Current Snapshot: MvccSnapshot[applied={T|T < 7315390988080005120}]
I20260806 02:42:07.968010 14137 mvcc.cc:205] Tried to move back new op lower bound from 7315390988080005120 to 7315390987902877696. Current Snapshot: MvccSnapshot[applied={T|T < 7315390988080005120}]
I20260806 02:42:07.968329 14136 mvcc.cc:205] Tried to move back new op lower bound from 7315390988080005120 to 7315390987902877696. Current Snapshot: MvccSnapshot[applied={T|T < 7315390988080005120}]
I20260806 02:42:07.976150 13336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:42:07.976318 13336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:42:07.979092 13336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe347837852d4eeb8ee882f2c8abdf03: Sending DeleteTablet for 3 replicas of tablet 609bc80c925c463e839df5313eb06e6e
I20260806 02:42:07.979771 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bc80c925c463e839df5313eb06e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:07 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:59820
I20260806 02:42:07.980319 13768 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bc80c925c463e839df5313eb06e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:07 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:38500
I20260806 02:42:07.981115 13977 tablet_service.cc:1514] Processing DeleteTablet for tablet 609bc80c925c463e839df5313eb06e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:07 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:39010
I20260806 02:42:07.982106 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 13453
I20260806 02:42:07.994078 14149 tablet_replica.cc:330] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: stopping tablet replica
I20260806 02:42:07.994800 14149 raft_consensus.cc:2229] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:42:07.995824 14149 raft_consensus.cc:2258] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:42:07.996248 14148 tablet_replica.cc:330] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: stopping tablet replica
I20260806 02:42:07.996625 14148 raft_consensus.cc:2229] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:42:07.998718 14148 raft_consensus.cc:2258] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:42:08.002919 14149 ts_tablet_manager.cc:1905] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:42:08.006572 13308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:44177 (error 108)
I20260806 02:42:08.006618 14149 ts_tablet_manager.cc:1918] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:42:08.006738 14149 log.cc:1196] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/609bc80c925c463e839df5313eb06e6e
I20260806 02:42:08.007107 14149 ts_tablet_manager.cc:1939] T 609bc80c925c463e839df5313eb06e6e P 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9: Deleting consensus metadata
I20260806 02:42:08.007169 14148 ts_tablet_manager.cc:1905] T 609bc80c925c463e839df5313eb06e6e P 3d3c5f69dea147389e04e794677a9d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:42:08.008286 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 13667
I20260806 02:42:08.008306 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:08.008006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:44177 (local address 127.0.0.1:59828)
0806 02:42:08.008132 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:08.008213 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:44177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 02:42:08.008479 13308 catalog_manager.cc:4584] TS 0970e30c47d040f789d91d08ef37fcd5 (127.28.191.1:44177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 609bc80c925c463e839df5313eb06e6e: Network error: Client connection negotiation failed: client connection to 127.28.191.1:44177: connect: Connection refused (error 111)
I20260806 02:42:08.012715 13315 catalog_manager.cc:4841] TS 0fd3cd1d5e2f4bee8d3e3dc70f4f68a9 (127.28.191.3:36229): tablet 609bc80c925c463e839df5313eb06e6e (table test-table [id=0911a18d285040e7b3e7f07a5960ef86]) successfully deleted
I20260806 02:42:08.027917 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 13879
W20260806 02:42:08.028545 14105 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:42:08.025392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:42813 (local address 127.0.0.1:38508)
0806 02:42:08.028379 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:08.028438 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:42813: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":2896}
I20260806 02:42:08.040525 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 13285
I20260806 02:42:08.058430 29436 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:42:08.403224 29436 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:42:08.403354 29436 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:42:08.748287 29436 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:42:08.748 UTC [12636] LOG: received smart shutdown request
2026-08-06 02:42:08.752 UTC [12636] LOG: background worker "logical replication launcher" (PID 12653) exited with exit code 1
2026-08-06 02:42:08.752 UTC [12645] LOG: shutting down
2026-08-06 02:42:08.903 UTC [12636] LOG: database system is shut down
2026-08-06T02:42:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45686 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:42:09 dist-test-slave-tfpf krb5kdc[14168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:42:09 dist-test-slave-tfpf krb5kdc[14168](info): set up 2 sockets
Aug 06 02:42:09 dist-test-slave-tfpf krb5kdc[14168](info): commencing operation
krb5kdc: starting...
W20260806 02:42:10.222659 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:42:10 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:42:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangeradmin_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangeradmin_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangerlookup_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangerlookup_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
I20260806 02:42:10.390937 29436 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklwD5TR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/postgres -l logfile start
2026-08-06 02:42:12.709 UTC [14212] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:42:12.709 UTC [14212] LOG: listening on IPv4 address "127.28.191.20", port 48781
2026-08-06 02:42:12.712 UTC [14212] LOG: listening on Unix socket "/tmp/.s.PGSQL.48781"
2026-08-06 02:42:12.732 UTC [14215] LOG: database system was shut down at 2026-08-06 02:42:11 UTC
2026-08-06 02:42:12.736 UTC [14212] LOG: database system is ready to accept connections
I20260806 02:42:13.849942 29436 mini_postgres.cc:96] Postgres bound to 48781
2026-08-06 02:42:13.857 UTC [14233] FATAL: database "slave" does not exist
127.28.191.20:48781 - accepting connections
I20260806 02:42:13.858433 29436 mini_ranger.cc:161] Starting Ranger...
I20260806 02:42:13.873776 29436 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:42:14.295327 29436 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:42:14.295446 29436 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin
2026-08-06 02:42:14,570 [I] DB FLAVOR :POSTGRES
2026-08-06 02:42:14,570 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:42:14,571 [I] Checking connection..
2026-08-06 02:42:14,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:42:14,836 [I] Checking connection passed.
2026-08-06 02:42:14,836 [I] --------- Verifying version history table ---------
2026-08-06 02:42:14,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:42:15,097 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:42:15,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:42:15,320 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:42:15,321 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:42:15,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:42:15.563 UTC [14314] WARNING: there is no transaction in progress
2026-08-06 02:42:15,569 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:42:15,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:42:15,813 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:42:15,814 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:42:15,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:42:16,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:42:16,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tfpf.c.gcp-upstream.internal','N') ;"
2026-08-06 02:42:16,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:42:16,840 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:42:16,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:42:17,104 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:42:17,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:42:17,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:42:17,598 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:42:17,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:42:18.480 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.496 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.511 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.525 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.540 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.557 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.609 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.616 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.629 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.638 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.648 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.661 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.671 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.677 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.683 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:18.689 UTC [14486] WARNING: there is no transaction in progress
2026-08-06 02:42:19,252 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:42:19,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:42:19,503 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:42:19,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:42:19,763 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:42:19,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:42:20,002 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:42:20,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:42:20,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:42:20,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tfpf.c.gcp-upstream.internal';"
2026-08-06 02:42:20,731 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:42:20,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:42:20,996 [I] DB_PATCHES have already been applied
I20260806 02:42:21.001744 29436 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklwD5TR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:42:21.001832 29436 mini_ranger.cc:193] Using host: 127.28.191.20
I20260806 02:42:21.006402 29436 mini_ranger.cc:227] Ranger admin URL: http://127.28.191.20:59085
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:42:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:42:22.302057 29436 mini_ranger.cc:160] Time spent starting Ranger: real 8.444s user 0.001s sys 0.011s
Aug 06, 2026 2:42:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:42:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:42:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:42:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:42:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:42:43.822185 29436 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.191.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.191.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangerkms_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.191.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/rangerkms_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.191.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.191.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.191.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/HTTP_127.28.191.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:42:43 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:42:43.911222 29436 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:42:43.921861 29436 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:42:44.663393 29436 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:42:44.663492 29436 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms
I20260806 02:42:44.663512 29436 mini_ranger_kms.cc:227] Using postgres at 127.28.191.20:48781
2026-08-06 02:42:44,831 [I] DB FLAVOR :POSTGRES
2026-08-06 02:42:44,831 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:42:44,832 [I] Checking connection
2026-08-06 02:42:44,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:42:45,079 [I] connection success
2026-08-06 02:42:45,079 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:42:45,079 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:42:45,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:42:45,322 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:42:45,323 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:42:45,323 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:42:45,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.191.20:48781/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:42:45,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:42:46.035534 29436 mini_ranger_kms.cc:310] Created kms service
I20260806 02:42:46.248214 29436 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:42:46.291628 29436 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:42:46 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:46.664152 29436 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:42:46.664310 29436 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklwD5TR/build/debug/bin/postgresql.jar:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklwD5TR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:42:46.664333 29436 mini_ranger_kms.cc:240] Using host: 127.28.191.20
I20260806 02:42:46.669275 29436 mini_ranger_kms.cc:276] Ranger KMS PID: 14764
I20260806 02:42:46.669358 29436 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.191.20:58521
Aug 06, 2026 2:42:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:42:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:42:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:42:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:42:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:42:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:42:49 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:49.334466 29436 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.423s user 0.001s sys 0.152s
I20260806 02:42:49.334617 29436 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:42:49.334718 29436 mini_ranger_kms.cc:401] 127.28.191.20:58521/kms/v1/keys
Aug 06 02:42:49 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
Aug 06 02:42:49 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.191.20@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
Aug 06 02:42:52 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:42:52.423900 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43021
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:43021
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:52.539098 14845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:52.539451 14845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:52.539494 14845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:52.543496 14845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:42:52.543550 14845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:52.543570 14845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:52.543586 14845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:42:52.543601 14845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:42:52.548975 14845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.191.20:58521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:43021
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:52.550168 14845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:52.551167 14845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:52.560041 14850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:52.560187 14845 server_base.cc:1034] running on GCE node
W20260806 02:42:52.560361 14851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:52.560039 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:52.560930 14845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:52.562065 14845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:52.563271 14845 hybrid_clock.cc:648] HybridClock initialized: now 1785984172563253 us; error 39 us; skew 500 ppm
Aug 06 02:42:52 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984172, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:52.566211 14845 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:42:52.567230 14845 webserver.cc:460] Webserver started at http://127.28.191.62:35077/ using document root <none> and password file <none>
I20260806 02:42:52.567775 14845 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:52.567831 14845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:52.568071 14845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:52 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984172, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:52.634714 14845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "baf304c102384d2d8bf6681c5317f8eb"
format_stamp: "Formatted at 2026-08-06 02:42:52 on dist-test-slave-tfpf"
server_key: "e74a133f29ad00d68498086c9fa17deb"
server_key_iv: "3fa5978820485f94519751a3ab9939e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:52.635357 14845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "baf304c102384d2d8bf6681c5317f8eb"
format_stamp: "Formatted at 2026-08-06 02:42:52 on dist-test-slave-tfpf"
server_key: "e74a133f29ad00d68498086c9fa17deb"
server_key_iv: "3fa5978820485f94519751a3ab9939e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:52.639487 14845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:42:52.657209 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:52.658903 14845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:42:52.659085 14845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "baf304c102384d2d8bf6681c5317f8eb"
format_stamp: "Formatted at 2026-08-06 02:42:52 on dist-test-slave-tfpf"
server_key: "e74a133f29ad00d68498086c9fa17deb"
server_key_iv: "3fa5978820485f94519751a3ab9939e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:52.659247 14845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:52.690548 14845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:52.694168 14845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:52.694392 14845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:52.703547 14943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:43021 every 8 connection(s)
I20260806 02:42:52.703562 14845 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:43021
I20260806 02:42:52.704941 14845 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:42:52.707808 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:52.712675 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 14845
I20260806 02:42:52.712824 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:42:52.713156 29436 external_mini_cluster.cc:1428] Setting key cd60391503872afcaeb22246b58b57c1
I20260806 02:42:52.713425 14944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Bootstrap starting.
I20260806 02:42:52.716272 14944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:52.717211 14944 log.cc:824] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:52.719488 14944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: No bootstrap required, opened a new log
Aug 06 02:42:52 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:52.723963 14944 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:52.724234 14944 raft_consensus.cc:374] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:52.724320 14944 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf304c102384d2d8bf6681c5317f8eb, State: Initialized, Role: FOLLOWER
I20260806 02:42:52.724821 14944 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:52.725041 14944 raft_consensus.cc:388] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:42:52.725132 14944 raft_consensus.cc:482] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:42:52.725235 14944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:52.726322 14944 raft_consensus.cc:504] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:52.726747 14944 leader_election.cc:302] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baf304c102384d2d8bf6681c5317f8eb; no voters:
I20260806 02:42:52.727125 14944 leader_election.cc:288] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:42:52.727490 14956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:42:52.728307 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:52.714641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.28.191.62:43021)
0806 02:42:52.715147 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:52.715166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:52.715207 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:42:52.716002 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:52.716024 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:52.716438 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:52.716836 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:52.717760 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:52.718820 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:52.719763 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:52.720228 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:52.720400 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:52.722966 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:52.722972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:52.722984 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:52.723010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:52.724845 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:52.725420 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:52.725426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:52.725430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:52.725533 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:52.725841 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:52.725844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:52.725846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:52.726264 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:52.726463 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:52.726835 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:42:52.727169 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":150,"threads_started":1}
I20260806 02:42:52.728406 14944 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:42:52.729419 14956 raft_consensus.cc:686] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 LEADER]: Becoming Leader. State: Replica: baf304c102384d2d8bf6681c5317f8eb, State: Running, Role: LEADER
I20260806 02:42:52.729815 14956 consensus_queue.cc:237] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:52.731138 14964 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader baf304c102384d2d8bf6681c5317f8eb. Latest consensus state: current_term: 1 leader_uuid: "baf304c102384d2d8bf6681c5317f8eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } } }
I20260806 02:42:52.731325 14964 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:52.731526 14944 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:42:52.731842 14981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:42:52.732954 14955 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baf304c102384d2d8bf6681c5317f8eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } } }
I20260806 02:42:52.735217 14955 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:52.739183 14981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:42:52.752180 14981 catalog_manager.cc:1347] Generated new cluster ID: 10396a55cc4a45ae83fa626109ba699f
I20260806 02:42:52.752259 14981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:42:52.766423 14981 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:42:52.768464 14981 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:42:52.777109 14981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Generated new TSK 0
I20260806 02:42:52.777972 14981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:42:53.505686 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:53.665947 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:53.666188 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:53.666230 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:53.670296 15015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:53.670347 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:53.670403 15015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:42:53.675848 15015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:53.676877 15015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:53.678189 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:53.689890 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:53.689998 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:53.690512 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:53.691464 15015 server_base.cc:1034] running on GCE node
I20260806 02:42:53.691954 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:53.692756 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:53.694015 15015 hybrid_clock.cc:648] HybridClock initialized: now 1785984173693975 us; error 62 us; skew 500 ppm
Aug 06 02:42:53 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:53.697223 15015 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:42:53.698360 15015 webserver.cc:460] Webserver started at http://127.28.191.1:43819/ using document root <none> and password file <none>
I20260806 02:42:53.699097 15015 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:53.699198 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:53.699512 15015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:53 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:53.721879 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "f587dc5c1c0347ae9124929fa875a662"
format_stamp: "Formatted at 2026-08-06 02:42:53 on dist-test-slave-tfpf"
server_key: "7063015b3dadf96c8b844c84a3e6688b"
server_key_iv: "8c38a805a582402fba608b33ff455883"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:53.722666 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "f587dc5c1c0347ae9124929fa875a662"
format_stamp: "Formatted at 2026-08-06 02:42:53 on dist-test-slave-tfpf"
server_key: "7063015b3dadf96c8b844c84a3e6688b"
server_key_iv: "8c38a805a582402fba608b33ff455883"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:53.727854 15015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:42:53.749234 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:53.750770 15015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:42:53.750909 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f587dc5c1c0347ae9124929fa875a662"
format_stamp: "Formatted at 2026-08-06 02:42:53 on dist-test-slave-tfpf"
server_key: "7063015b3dadf96c8b844c84a3e6688b"
server_key_iv: "8c38a805a582402fba608b33ff455883"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:53.751061 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:53.805574 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:53.809953 15015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:53.810179 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:53.810876 15015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:53.811818 15015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:53.811890 15015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:53.811968 15015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:53.812021 15015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:53.823135 15015 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42867
I20260806 02:42:53.823174 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42867 every 8 connection(s)
I20260806 02:42:53.824239 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:42:53.834285 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15015
Aug 06 02:42:53 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:53.834419 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:42:53.834769 29436 external_mini_cluster.cc:1428] Setting key 5a492b711787d346a1ae66ae89cc42a1
I20260806 02:42:53.839779 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:53.826392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:43647 (local address 127.28.191.62:43021)
0806 02:42:53.826662 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:53.826667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:53.827479 (+ 812us) server_negotiation.cc:407] Connection header received
0806 02:42:53.828570 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:53.828574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:53.828636 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:53.828755 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:53.830061 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:53.830768 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:53.831943 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:53.832204 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:53.832316 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:53.835031 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:53.835036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:53.835038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:53.835068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:53.837023 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:53.837756 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:53.837762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:53.837767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:53.837866 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:53.838325 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:53.838331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:53.838335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:53.838593 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:53.838671 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:53.839455 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 02:42:53.839593 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260806 02:42:53.841861 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:53.826847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.1:43647)
0806 02:42:53.827316 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:53.827370 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:53.828360 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:53.828945 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:53.828952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:53.829340 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:53.829832 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:53.829847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:53.830903 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:53.830908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:53.831802 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:53.831812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:53.832362 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:53.832368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:53.832552 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:53.833186 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:53.833203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:53.834809 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:53.837212 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:53.837219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:53.837241 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:53.837627 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:53.837986 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:53.837992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:53.837997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:53.838207 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:53.838765 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:53.838775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:53.839159 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 02:42:53.840311 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0806 02:42:53.840741 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
I20260806 02:42:53.843523 15205 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:53.843779 15205 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:53.844419 15205 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:53.846280 14891 ts_manager.cc:194] Registered new tserver with Master: f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867)
I20260806 02:42:53.847823 14891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:43647
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:42:53.884361 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:42:53 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:42:54.049041 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:54.049559 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:54.049702 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:54.055962 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:54.056052 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:54.056149 15211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:42:54.064297 15211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:54.066020 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:54.067464 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:54.080950 15220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.081362 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.081616 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:54.082967 15211 server_base.cc:1034] running on GCE node
I20260806 02:42:54.083551 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:54.084323 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:54.085480 15211 hybrid_clock.cc:648] HybridClock initialized: now 1785984174085414 us; error 82 us; skew 500 ppm
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:54.089190 15211 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:42:54.090276 15211 webserver.cc:460] Webserver started at http://127.28.191.2:39393/ using document root <none> and password file <none>
I20260806 02:42:54.090963 15211 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:54.091048 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:54.091285 15211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:54.114038 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "8d13cebf17284aefa5bc8ca736d9132c"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "89290429be098ecc15e75ceae4dc280b"
server_key_iv: "deca1a1cc490a4246f72eaa56ac2c844"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.115159 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "8d13cebf17284aefa5bc8ca736d9132c"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "89290429be098ecc15e75ceae4dc280b"
server_key_iv: "deca1a1cc490a4246f72eaa56ac2c844"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.120396 15211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:42:54.148823 15231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.150609 15211 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.000s sys 0.002s
I20260806 02:42:54.150738 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "8d13cebf17284aefa5bc8ca736d9132c"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "89290429be098ecc15e75ceae4dc280b"
server_key_iv: "deca1a1cc490a4246f72eaa56ac2c844"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.150869 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:54.177122 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:54.180807 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:54.181098 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:54.181843 15211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:54.182991 15211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:54.183138 15211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.183248 15211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:54.183316 15211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.200901 15211 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:32791
I20260806 02:42:54.201099 15402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:32791 every 8 connection(s)
I20260806 02:42:54.202189 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:42:54.207149 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15211
I20260806 02:42:54.207279 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:42:54.207620 29436 external_mini_cluster.cc:1428] Setting key a3032e039423a4e63fcd76c0cef60221
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:54.220862 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:54.204288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:35347 (local address 127.28.191.62:43021)
0806 02:42:54.204452 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:54.204458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:54.205519 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 02:42:54.206822 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:54.206826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:54.206882 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:54.206963 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:54.209488 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.210296 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.212044 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.212409 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.212488 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:54.215861 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:54.215865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:54.215868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:54.215895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:54.217994 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:54.218652 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:54.218657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:54.218660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:54.218771 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:54.219295 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:54.219301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:54.219304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:54.219583 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:54.219704 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:54.220419 (+ 715us) server_negotiation.cc:299] Negotiation successful
0806 02:42:54.220586 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:42:54.221805 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:54.204772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.2:35347)
0806 02:42:54.205356 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:54.205414 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:54.206570 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:54.208264 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:54.208274 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:54.208624 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:54.209317 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:54.209337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.210479 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.210482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:54.211895 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:54.211907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.212687 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.212691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:54.212919 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:54.213652 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:54.213673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:54.215658 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:54.218147 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:54.218153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:54.218170 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:54.218514 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:54.218902 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:54.218905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:54.218908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:54.219109 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:54.219718 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:54.219726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:54.220069 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 02:42:54.220318 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 02:42:54.220673 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":197,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:42:54.233181 15403 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:54.233467 15403 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:54.234148 15403 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
I20260806 02:42:54.235414 14891 ts_manager.cc:194] Registered new tserver with Master: 8d13cebf17284aefa5bc8ca736d9132c (127.28.191.2:32791)
I20260806 02:42:54.236289 14891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:35347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:42:54.253742 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
W20260806 02:42:54.444805 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:54.445255 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:54.445371 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:54.451169 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:54.451252 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:54.451371 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:42:54.458150 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:54.459420 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:54.460760 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:54.480798 15416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.481009 15415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.480801 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:54.490487 15410 server_base.cc:1034] running on GCE node
I20260806 02:42:54.491070 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:54.491894 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:54.495172 15410 hybrid_clock.cc:648] HybridClock initialized: now 1785984174494732 us; error 446 us; skew 500 ppm
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:54.499969 15410 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:42:54.500881 15410 webserver.cc:460] Webserver started at http://127.28.191.3:35113/ using document root <none> and password file <none>
I20260806 02:42:54.501672 15410 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:54.501823 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:54.502118 15410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:54.525512 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "76b4a0e77fc847199f090a0457533a64"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "4bd60c0dfc23107d118541ce11aa38b4"
server_key_iv: "7772fa286688d296080ed7ada43a3aab"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.526434 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "76b4a0e77fc847199f090a0457533a64"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "4bd60c0dfc23107d118541ce11aa38b4"
server_key_iv: "7772fa286688d296080ed7ada43a3aab"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.532398 15410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:42:54.565462 15426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.567318 15410 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260806 02:42:54.567472 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "76b4a0e77fc847199f090a0457533a64"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "4bd60c0dfc23107d118541ce11aa38b4"
server_key_iv: "7772fa286688d296080ed7ada43a3aab"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.567615 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:54.592170 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:54.604000 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:54.604206 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:54.604874 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:54.605970 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:54.606020 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.606083 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:54.606106 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.617801 15410 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:35621
I20260806 02:42:54.622884 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:42:54.617866 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:35621 every 8 connection(s)
I20260806 02:42:54.627866 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15410
I20260806 02:42:54.628008 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:42:54.628352 29436 external_mini_cluster.cc:1428] Setting key 61fc2627d6093a573baf6be43b80129e
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:54.649718 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:54.626080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:50707 (local address 127.28.191.62:43021)
0806 02:42:54.626228 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:54.626232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:54.633618 (+ 7386us) server_negotiation.cc:407] Connection header received
0806 02:42:54.635006 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:54.635010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:54.635062 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:54.635111 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:54.636748 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.637618 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.639298 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.639648 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.639718 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:54.645291 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:54.645295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:54.645298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:54.645324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:54.647293 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:54.647911 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:54.647915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:54.647917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:54.648003 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:54.648426 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:54.648430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:54.648432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:54.648655 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:54.648726 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:54.649426 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 02:42:54.649558 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:42:54.651600 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:54.626558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.3:50707)
0806 02:42:54.633448 (+ 6890us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:54.633520 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:54.634760 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:54.635539 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:54.635549 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:54.635914 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:54.636570 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:54.636587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.637777 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.637780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:54.639159 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:54.639169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:54.639909 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:54.639912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:54.640113 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:54.640785 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:54.643292 (+ 2507us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:54.645108 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:54.647434 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:54.647440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:54.647455 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:54.647796 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:54.648113 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:54.648115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:54.648117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:54.648304 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:54.648809 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:54.648815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:54.649137 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 02:42:54.650147 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0806 02:42:54.650529 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6629,"thread_start_us":185,"threads_started":1}
I20260806 02:42:54.653005 15586 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:54.653258 15586 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:54.653820 15586 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:54.656222 14891 ts_manager.cc:194] Registered new tserver with Master: 76b4a0e77fc847199f090a0457533a64 (127.28.191.3:35621)
I20260806 02:42:54.656955 14891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:50707
I20260806 02:42:54.667152 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:42:54.667889 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15015
I20260806 02:42:54.696812 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15211
I20260806 02:42:54.706488 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15410
I20260806 02:42:54.726351 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 14845
I20260806 02:42:54.740355 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43021
--webserver_interface=127.28.191.62
--webserver_port=35077
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:43021
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:54.852880 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:54.853153 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:54.853190 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:54.857138 15589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:42:54.857187 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:54.857206 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:54.857221 15589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:42:54.857236 15589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:42:54.862308 15589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:43021
--ranger_config_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:54.863168 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:54.864087 15589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:54.872610 15597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.872649 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:54.872665 15595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:54.872793 15589 server_base.cc:1034] running on GCE node
I20260806 02:42:54.873544 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:54.874572 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:54.875772 15589 hybrid_clock.cc:648] HybridClock initialized: now 1785984174875759 us; error 24 us; skew 500 ppm
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:54.878656 15589 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:42:54.879527 15589 webserver.cc:460] Webserver started at http://127.28.191.62:35077/ using document root <none> and password file <none>
I20260806 02:42:54.880023 15589 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:54.880090 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:42:54 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984174, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:54.962029 15589 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:42:54.964188 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:54.965557 15589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:42:54.965724 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "baf304c102384d2d8bf6681c5317f8eb"
format_stamp: "Formatted at 2026-08-06 02:42:52 on dist-test-slave-tfpf"
server_key: "e74a133f29ad00d68498086c9fa17deb"
server_key_iv: "3fa5978820485f94519751a3ab9939e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:54.966624 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:54.982928 15589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:54.985920 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:54.986130 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:54.993558 15589 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:43021
I20260806 02:42:54.993573 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:43021 every 8 connection(s)
I20260806 02:42:54.994673 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:42:54.998215 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15589
I20260806 02:42:54.998680 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:42867
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:43021
--webserver_port=43819
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:42:54.999156 15699 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:42:55.000795 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Bootstrap starting.
I20260806 02:42:55.017199 15699 log.cc:824] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:55.023923 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:42:55.024339 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Bootstrap complete.
I20260806 02:42:55.028470 15699 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:55.028900 15699 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf304c102384d2d8bf6681c5317f8eb, State: Initialized, Role: FOLLOWER
I20260806 02:42:55.029485 15699 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:55.029642 15699 raft_consensus.cc:388] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:42:55.029736 15699 raft_consensus.cc:482] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:42:55.029838 15699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:42:55.031658 15699 raft_consensus.cc:504] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:55.031988 15699 leader_election.cc:302] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baf304c102384d2d8bf6681c5317f8eb; no voters:
I20260806 02:42:55.032356 15699 leader_election.cc:288] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:42:55.032447 15704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:42:55.032672 15704 raft_consensus.cc:686] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [term 2 LEADER]: Becoming Leader. State: Replica: baf304c102384d2d8bf6681c5317f8eb, State: Running, Role: LEADER
I20260806 02:42:55.033017 15704 consensus_queue.cc:237] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } }
I20260806 02:42:55.033613 15699 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:42:55.034365 15705 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "baf304c102384d2d8bf6681c5317f8eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } } }
I20260806 02:42:55.034492 15705 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:55.035192 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:42:55.035229 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader baf304c102384d2d8bf6681c5317f8eb. Latest consensus state: current_term: 2 leader_uuid: "baf304c102384d2d8bf6681c5317f8eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf304c102384d2d8bf6681c5317f8eb" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43021 } } }
I20260806 02:42:55.035387 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:42:55.038530 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:42:55.039069 15699 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:42:55.039197 15715 catalog_manager.cc:1259] Loaded cluster ID: 10396a55cc4a45ae83fa626109ba699f
I20260806 02:42:55.039252 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:42:55.041636 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:42:55.042300 15715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Loaded TSK: 0
I20260806 02:42:55.043498 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:42:55.141532 15702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:55.141803 15702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:55.141903 15702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:55.146013 15702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:55.146095 15702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:55.146205 15702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:42:55.153389 15702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:42867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=43819
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:55.155041 15702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:55.156369 15702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:55.166672 15748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:55.166797 15747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:55.167141 15702 server_base.cc:1034] running on GCE node
W20260806 02:42:55.166949 15751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:55.167793 15702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:55.168466 15702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:55.169678 15702 hybrid_clock.cc:648] HybridClock initialized: now 1785984175169654 us; error 42 us; skew 500 ppm
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:55.172947 15702 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:42:55.174149 15702 webserver.cc:460] Webserver started at http://127.28.191.1:43819/ using document root <none> and password file <none>
I20260806 02:42:55.174886 15702 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:55.175005 15702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:55.200649 15702 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260806 02:42:55.206102 15761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.207940 15702 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.002s
I20260806 02:42:55.208115 15702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f587dc5c1c0347ae9124929fa875a662"
format_stamp: "Formatted at 2026-08-06 02:42:53 on dist-test-slave-tfpf"
server_key: "7063015b3dadf96c8b844c84a3e6688b"
server_key_iv: "8c38a805a582402fba608b33ff455883"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:55.208881 15702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:55.222227 15702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:55.225574 15702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:55.225802 15702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:55.226636 15702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:55.227771 15702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:55.227833 15702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.227875 15702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:55.227927 15702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.239223 15702 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:42867
I20260806 02:42:55.240434 15702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:42:55.241493 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:42867 every 8 connection(s)
I20260806 02:42:55.247383 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15702
I20260806 02:42:55.247941 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:32791
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:43021
--webserver_port=39393
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:55.262202 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.243136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.1:37149)
0806 02:42:55.243945 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:55.243998 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:55.244850 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:55.246061 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:55.246069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:55.246427 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:55.246890 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.246905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.248442 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.248447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.249441 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.249452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.250687 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.250693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.250996 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.251778 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:55.251806 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:55.255132 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:55.258229 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.258237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.258254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.258666 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.259129 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.259132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.259134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.259310 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.260149 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:55.260156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:55.260477 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 02:42:55.260709 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:42:55.261097 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":145,"threads_started":1}
I20260806 02:42:55.264225 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.242983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:37149 (local address 127.28.191.62:43021)
0806 02:42:55.243642 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:55.243683 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:55.244141 (+ 458us) server_negotiation.cc:407] Connection header received
0806 02:42:55.245329 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:55.245340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:55.245648 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:55.245974 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:55.247075 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.248275 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.249585 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.250223 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.250416 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.255398 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:55.255407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:55.255427 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:55.255466 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:55.258049 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.258834 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.258841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.258846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.259014 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.259410 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.259415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.259419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.259953 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:55.260090 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:55.262791 (+ 2701us) server_negotiation.cc:299] Negotiation successful
0806 02:42:55.263146 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":134,"threads_started":1}
I20260806 02:42:55.267724 15936 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:55.267987 15936 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.268635 15936 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.271133 15936 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:43021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:42:55.271370 15936 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.271646 15936 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:55.272709 15936 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.272957 15936 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.273814 15936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:42:55.412374 15941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:55.412619 15941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:55.412660 15941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:55.417281 15941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:55.417336 15941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:55.417419 15941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:42:55.424192 15941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:32791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=39393
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:55.425514 15941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:55.426800 15941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:55.437805 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:55.438050 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:55.438215 15941 server_base.cc:1034] running on GCE node
W20260806 02:42:55.438297 15952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:55.438745 15941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:55.439489 15941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:55.440694 15941 hybrid_clock.cc:648] HybridClock initialized: now 1785984175440671 us; error 41 us; skew 500 ppm
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:55.444818 15941 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:42:55.445727 15941 webserver.cc:460] Webserver started at http://127.28.191.2:39393/ using document root <none> and password file <none>
I20260806 02:42:55.446341 15941 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:55.446445 15941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:55.470389 15941 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260806 02:42:55.473461 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.478231 15941 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260806 02:42:55.478442 15941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "8d13cebf17284aefa5bc8ca736d9132c"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "89290429be098ecc15e75ceae4dc280b"
server_key_iv: "deca1a1cc490a4246f72eaa56ac2c844"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:55.479226 15941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:55.494108 15941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:55.497740 15941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:55.498018 15941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:55.498805 15941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:55.499933 15941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:55.500043 15941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.500157 15941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:55.500229 15941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.512147 15941 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:32791
I20260806 02:42:55.513758 15941 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:42:55.518975 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:32791 every 8 connection(s)
I20260806 02:42:55.519474 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 15941
I20260806 02:42:55.519995 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.191.20:58521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:35621
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:43021
--webserver_port=35113
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:55.538908 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57415 (local address 127.28.191.62:43021)
0806 02:42:55.517463 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:55.517467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:55.518630 (+ 1163us) server_negotiation.cc:407] Connection header received
0806 02:42:55.519725 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:55.519728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:55.519785 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:55.519875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:55.521237 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.521984 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.523744 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.524103 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.524184 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.533830 (+ 9646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:55.533837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:55.533842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:55.533914 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:55.535884 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.536859 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.536866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.536869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.536975 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.537513 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.537517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.537519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.537761 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:55.537847 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:55.538584 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:42:55.538732 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:42:55.540838 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.517863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.2:57415)
0806 02:42:55.518459 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:55.518522 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:55.519479 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:55.520093 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:55.520104 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:55.520450 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:55.521042 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.521056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.522157 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.522163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.523589 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.523602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.524395 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.524399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.524605 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.528774 (+ 4169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:55.528797 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:55.533544 (+ 4747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:55.536109 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.536119 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.536140 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.536701 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.537151 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.537157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.537162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.537378 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.537917 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:55.537924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:55.538293 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 02:42:55.539371 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0806 02:42:55.539736 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":186,"threads_started":1}
I20260806 02:42:55.542351 16117 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:55.542606 16117 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.543067 16117 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.544625 16117 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:43021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:42:55.544747 16117 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.544901 16117 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:55.545746 16117 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.545925 16117 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.546763 16117 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:42:55.664436 16120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:55.664783 16120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:55.664866 16120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:55.669102 16120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:55.669214 16120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:55.669337 16120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:42:55.675182 16120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.191.20:58521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:35621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=35113
--tserver_master_addrs=127.28.191.62:43021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:55.676389 16120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:55.677491 16120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:55.688719 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:55.691877 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:55.693243 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:55.698207 16120 server_base.cc:1034] running on GCE node
I20260806 02:42:55.698698 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:55.699402 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:55.700565 16120 hybrid_clock.cc:648] HybridClock initialized: now 1785984175700535 us; error 45 us; skew 500 ppm
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:55.703629 16120 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:42:55.704687 16120 webserver.cc:460] Webserver started at http://127.28.191.3:35113/ using document root <none> and password file <none>
I20260806 02:42:55.705389 16120 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:55.705507 16120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:55.723069 16120 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:42:55.725768 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.727072 16120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:42:55.727200 16120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "76b4a0e77fc847199f090a0457533a64"
format_stamp: "Formatted at 2026-08-06 02:42:54 on dist-test-slave-tfpf"
server_key: "4bd60c0dfc23107d118541ce11aa38b4"
server_key_iv: "7772fa286688d296080ed7ada43a3aab"
server_key_version: "kuduclusterkey@0"
I20260806 02:42:55.727766 16120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:55.754045 16120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:55.757632 16120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:55.757854 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:55.758598 16120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:42:55.759533 16120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:42:55.759611 16120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.759680 16120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:42:55.759711 16120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:55.771132 16120 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:35621
I20260806 02:42:55.771196 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:35621 every 8 connection(s)
I20260806 02:42:55.772568 16120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:42:55.782848 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 16120
Aug 06 02:42:55 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984175, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:42:55.790792 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.774483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:32993 (local address 127.28.191.62:43021)
0806 02:42:55.774837 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:55.774842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:55.775651 (+ 809us) server_negotiation.cc:407] Connection header received
0806 02:42:55.776963 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:55.776968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:55.777037 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:55.777174 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:55.778633 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.779448 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.781280 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.781626 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.781716 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.786401 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:55.786406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:55.786408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:55.786437 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:55.788240 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.788867 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.788870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.788872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.788968 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.789446 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.789450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.789451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.789678 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:55.789758 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:55.790501 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 02:42:55.790636 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260806 02:42:55.792598 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.774909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:43021 (local address 127.28.191.3:32993)
0806 02:42:55.775487 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:55.775546 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:55.776684 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:55.777368 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:55.777377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:55.777734 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:55.778434 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.778457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.779627 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.779631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.781120 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:55.781133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.781931 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.781935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:55.782142 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.784306 (+ 2164us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:55.784327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:55.786202 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:55.788378 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.788384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.788399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.788738 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.789118 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:55.789121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:55.789123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:55.789321 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:55.789847 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:55.789854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:55.790182 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 02:42:55.791170 (+ 988us) client_negotiation.cc:240] Negotiation successful
0806 02:42:55.791528 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":139,"threads_started":1}
I20260806 02:42:55.795146 16326 heartbeater.cc:347] Connected to a master server at 127.28.191.62:43021
I20260806 02:42:55.795408 16326 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.795958 16326 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.797721 16326 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:43021 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:42:55.797863 16326 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.798058 16326 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:55.799027 16326 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:55.799276 16326 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
W20260806 02:42:55.800334 16326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:42:55.802196 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:55.783992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.28.191.62:43021)
0806 02:42:55.785207 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:55.785267 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:55.785727 (+ 460us) server_negotiation.cc:407] Connection header received
0806 02:42:55.785769 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:55.785772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:55.786787 (+ 1015us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:55.786902 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:55.792925 (+ 6023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.793734 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.795370 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:55.795730 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:55.795843 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:55.798449 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:55.798455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:55.798458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:55.798486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:55.800439 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.800924 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.800926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.800928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.801002 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:55.801334 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:55.801336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:55.801337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:55.801502 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:55.801645 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:55.801906 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:42:55.802020 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":85,"threads_started":1}
Aug 06 02:42:56 dist-test-slave-tfpf krb5kdc[14168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:56.274106 15936 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:56.274423 15936 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:56.275784 15636 ts_manager.cc:194] Registered new tserver with Master: f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867)
I20260806 02:42:56.277631 15636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:37149
Aug 06 02:42:56 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:56.546988 16117 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:56.547304 16117 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:56.548139 15634 ts_manager.cc:194] Registered new tserver with Master: 8d13cebf17284aefa5bc8ca736d9132c (127.28.191.2:32791)
I20260806 02:42:56.548941 15634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:57415
Aug 06 02:42:56 dist-test-slave-tfpf krb5kdc[14168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984176, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for HTTP/127.28.191.20@KRBTEST.COM
I20260806 02:42:56.800576 16326 heartbeater.cc:464] Registering TS with master...
I20260806 02:42:56.800941 16326 heartbeater.cc:511] Master 127.28.191.62:43021 requested a full tablet report, sending...
I20260806 02:42:56.801750 15634 ts_manager.cc:194] Registered new tserver with Master: 76b4a0e77fc847199f090a0457533a64 (127.28.191.3:35621)
I20260806 02:42:56.802539 15634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:32993
I20260806 02:42:56.807361 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:42:56.819205 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.809013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.28.191.62:43021)
0806 02:42:56.809318 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:56.809324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:56.809595 (+ 271us) server_negotiation.cc:407] Connection header received
0806 02:42:56.809749 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:56.809752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:56.809805 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:56.809900 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:56.810586 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.811310 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.812543 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.812891 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.812993 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.814878 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:56.814891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:56.814894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:56.814922 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:56.816828 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:56.817409 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:56.817413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:56.817415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:56.817512 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:56.817846 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:56.817849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:56.817851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:56.818084 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:56.818223 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:56.818882 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 02:42:56.819013 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":132,"threads_started":1}
I20260806 02:42:56.823714 15634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:42:56.828889 15634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:42:56.859894 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.846920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32791 (local address 127.0.0.1:43002)
0806 02:42:56.850850 (+ 3930us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:56.850908 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:56.851079 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:56.853883 (+ 2804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:56.853891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:56.853923 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:56.854086 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.854095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.855586 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.855589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.857832 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.857844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.859268 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.859273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.859391 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.859442 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:42:56.859568 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:42:56.859709 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3738,"thread_start_us":108,"threads_started":1}
I20260806 02:42:56.861104 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.847638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.28.191.2:32791)
0806 02:42:56.853313 (+ 5675us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:56.853322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:56.853353 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:42:56.853448 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:56.853455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:56.853699 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:56.853891 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:56.854267 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.855443 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.857981 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.858849 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.858987 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.859179 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:56.860805 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0806 02:42:56.860920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5539,"thread_start_us":138,"threads_started":1}
I20260806 02:42:56.864452 16024 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 (DEFAULT_TABLE table=test-table [id=4cd7afa5afdd4c19b026ab17818b26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:56.865306 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.848484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:35621 (local address 127.0.0.1:60540)
0806 02:42:56.854198 (+ 5714us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:56.854216 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:56.854317 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:56.855245 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:56.855248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:56.855268 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:56.855375 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.855383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.862481 (+ 7098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.862485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.863592 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.863606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.864849 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.864853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.864953 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.864980 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:42:56.865079 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:42:56.865165 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5633,"thread_start_us":90,"threads_started":1}
I20260806 02:42:56.865712 16024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc1d919804f4a87b29c7331a25f1ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:56.865836 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.848927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.28.191.3:35621)
0806 02:42:56.854601 (+ 5674us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:56.854608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:56.854634 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:42:56.854713 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:56.854718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:56.854913 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:56.855080 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:56.861166 (+ 6086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.862308 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.863763 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.864534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.864646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.864781 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:56.865598 (+ 817us) server_negotiation.cc:299] Negotiation successful
0806 02:42:56.865700 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5556,"thread_start_us":120,"threads_started":1}
I20260806 02:42:56.868330 16215 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 (DEFAULT_TABLE table=test-table [id=4cd7afa5afdd4c19b026ab17818b26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:56.869403 16215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc1d919804f4a87b29c7331a25f1ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:56.869801 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.849740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42867 (local address 127.0.0.1:34184)
0806 02:42:56.855619 (+ 5879us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:56.855634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:56.855724 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:56.864665 (+ 8941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:56.864668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:56.864690 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:56.864791 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.864799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.866296 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.866300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.867497 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:56.867511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.869414 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.869421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:56.869517 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.869543 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:42:56.869595 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:42:56.869663 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5813,"thread_start_us":97,"threads_started":1}
I20260806 02:42:56.870549 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:56.850172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.28.191.1:42867)
0806 02:42:56.863953 (+ 13781us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:56.863962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:56.864004 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:42:56.864091 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:56.864097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:56.864306 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:56.864470 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:56.864965 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.866139 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.867659 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:56.869266 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:56.870058 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:56.870211 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:56.870261 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 02:42:56.870360 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13651,"thread_start_us":98,"threads_started":1}
I20260806 02:42:56.873045 15837 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 (DEFAULT_TABLE table=test-table [id=4cd7afa5afdd4c19b026ab17818b26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:42:56.874178 15837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc1d919804f4a87b29c7331a25f1ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:56.889132 16378 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Bootstrap starting.
I20260806 02:42:56.889430 16375 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Bootstrap starting.
I20260806 02:42:56.891423 16378 tablet_bootstrap.cc:654] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:56.891785 16375 tablet_bootstrap.cc:654] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:56.892328 16378 log.cc:824] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:56.892697 16375 log.cc:824] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:56.895133 16375 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: No bootstrap required, opened a new log
I20260806 02:42:56.895438 16375 ts_tablet_manager.cc:1397] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:42:56.897467 16376 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Bootstrap starting.
I20260806 02:42:56.899667 16376 tablet_bootstrap.cc:654] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Neither blocks nor log segments found. Creating new log.
I20260806 02:42:56.900218 16375 raft_consensus.cc:348] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.900440 16376 log.cc:824] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Log is configured to *not* fsync() on all Append() calls
I20260806 02:42:56.900668 16375 raft_consensus.cc:374] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:56.900805 16375 raft_consensus.cc:729] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d13cebf17284aefa5bc8ca736d9132c, State: Initialized, Role: FOLLOWER
I20260806 02:42:56.901455 16375 consensus_queue.cc:260] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.902734 16375 ts_tablet_manager.cc:1428] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:42:56.903211 16117 heartbeater.cc:503] Master 127.28.191.62:43021 was elected leader, sending a full tablet report...
I20260806 02:42:56.909440 16378 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: No bootstrap required, opened a new log
I20260806 02:42:56.909751 16378 ts_tablet_manager.cc:1397] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260806 02:42:56.914443 16378 raft_consensus.cc:348] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.914907 16378 raft_consensus.cc:374] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:56.915033 16378 raft_consensus.cc:729] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4a0e77fc847199f090a0457533a64, State: Initialized, Role: FOLLOWER
I20260806 02:42:56.915601 16378 consensus_queue.cc:260] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.919163 16378 ts_tablet_manager.cc:1428] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 02:42:56.920233 16326 heartbeater.cc:503] Master 127.28.191.62:43021 was elected leader, sending a full tablet report...
I20260806 02:42:56.924746 16376 tablet_bootstrap.cc:492] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: No bootstrap required, opened a new log
I20260806 02:42:56.924995 16376 ts_tablet_manager.cc:1397] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20260806 02:42:56.929133 16376 raft_consensus.cc:348] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.929684 16376 raft_consensus.cc:374] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:42:56.929788 16376 raft_consensus.cc:729] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f587dc5c1c0347ae9124929fa875a662, State: Initialized, Role: FOLLOWER
I20260806 02:42:56.930311 16376 consensus_queue.cc:260] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:56.932577 16376 ts_tablet_manager.cc:1428] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260806 02:42:56.933007 15936 heartbeater.cc:503] Master 127.28.191.62:43021 was elected leader, sending a full tablet report...
W20260806 02:42:56.996295 15937 tablet.cc:2369] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:42:57.017671 16118 tablet.cc:2369] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:42:57.024435 16328 tablet.cc:2369] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:42:57.116326 16391 raft_consensus.cc:482] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:42:57.116593 16391 raft_consensus.cc:504] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:57.118510 16391 leader_election.cc:288] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d13cebf17284aefa5bc8ca736d9132c (127.28.191.2:32791), f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867)
I20260806 02:42:57.126120 16388 raft_consensus.cc:482] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:42:57.126310 16388 raft_consensus.cc:504] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:57.127658 16388 leader_election.cc:288] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b4a0e77fc847199f090a0457533a64 (127.28.191.3:35621), f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867)
I20260806 02:42:57.129225 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.119958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60403 (local address 127.28.191.1:42867)
0806 02:42:57.120092 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:57.120097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:57.120437 (+ 340us) server_negotiation.cc:407] Connection header received
0806 02:42:57.120758 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:57.120761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:57.120819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:57.120867 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:57.122801 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.123529 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.126772 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.127299 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.127395 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.127449 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:57.128964 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0806 02:42:57.129057 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:42:57.129910 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.119585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42867 (local address 127.28.191.3:60403)
0806 02:42:57.120366 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.120377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:57.120532 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:57.121300 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:57.121303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:57.121319 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:57.121457 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.121463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.124904 (+ 3441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.124910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.126304 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.126315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.127560 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.127563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.127664 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.127695 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:42:57.129642 (+ 1947us) client_negotiation.cc:240] Negotiation successful
0806 02:42:57.129737 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":93,"threads_started":1}
I20260806 02:42:57.130481 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.119117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:47505 (local address 127.28.191.2:32791)
0806 02:42:57.119250 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:57.119255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:57.120254 (+ 999us) server_negotiation.cc:407] Connection header received
0806 02:42:57.121021 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:57.121025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:57.121109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:57.121162 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:57.121858 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.122659 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.127828 (+ 5169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.128433 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.128544 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.128590 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:57.130253 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0806 02:42:57.130335 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:42:57.131059 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.119003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32791 (local address 127.28.191.3:47505)
0806 02:42:57.120157 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.120172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:57.120624 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:57.121603 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:57.121605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:57.121623 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:57.121736 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.121741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.123682 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.123686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.126635 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.126645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.128683 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.128686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.128783 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.128807 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:42:57.130819 (+ 2012us) client_negotiation.cc:240] Negotiation successful
0806 02:42:57.130902 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":115,"threads_started":1}
I20260806 02:42:57.131744 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "76b4a0e77fc847199f090a0457533a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f587dc5c1c0347ae9124929fa875a662" is_pre_election: true
I20260806 02:42:57.132083 15864 raft_consensus.cc:2454] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4a0e77fc847199f090a0457533a64 in term 0.
I20260806 02:42:57.132149 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "76b4a0e77fc847199f090a0457533a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d13cebf17284aefa5bc8ca736d9132c" is_pre_election: true
I20260806 02:42:57.132339 16045 raft_consensus.cc:2454] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4a0e77fc847199f090a0457533a64 in term 0.
I20260806 02:42:57.132653 16150 leader_election.cc:302] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b4a0e77fc847199f090a0457533a64, f587dc5c1c0347ae9124929fa875a662; no voters:
I20260806 02:42:57.132939 16391 raft_consensus.cc:2781] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:42:57.133039 16391 raft_consensus.cc:482] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:42:57.133116 16391 raft_consensus.cc:3037] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:57.134570 16391 raft_consensus.cc:504] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:57.134946 16391 leader_election.cc:288] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [CANDIDATE]: Term 1 election: Requested vote from peers 8d13cebf17284aefa5bc8ca736d9132c (127.28.191.2:32791), f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867)
I20260806 02:42:57.135398 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "76b4a0e77fc847199f090a0457533a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d13cebf17284aefa5bc8ca736d9132c"
I20260806 02:42:57.135509 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "76b4a0e77fc847199f090a0457533a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f587dc5c1c0347ae9124929fa875a662"
I20260806 02:42:57.135522 16045 raft_consensus.cc:3037] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:57.135622 15864 raft_consensus.cc:3037] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:42:57.136147 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.128125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:53685 (local address 127.28.191.3:35621)
0806 02:42:57.128244 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:57.128248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:57.130853 (+ 2605us) server_negotiation.cc:407] Connection header received
0806 02:42:57.131047 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:57.131052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:57.131108 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:57.131164 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:57.131606 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.132292 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.134840 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.135458 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.135555 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.135624 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:57.135971 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 02:42:57.136051 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:42:57.136528 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.128062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:35621 (local address 127.28.191.2:53685)
0806 02:42:57.130768 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.130782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:57.130924 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:57.131309 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:57.131312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:57.131331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:57.131470 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.131478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.132597 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.132602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.133814 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.133828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.135731 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.135735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.135834 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.135857 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:42:57.136292 (+ 435us) client_negotiation.cc:240] Negotiation successful
0806 02:42:57.136380 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2639,"thread_start_us":100,"threads_started":1}
I20260806 02:42:57.136883 16045 raft_consensus.cc:2454] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4a0e77fc847199f090a0457533a64 in term 1.
I20260806 02:42:57.136879 15864 raft_consensus.cc:2454] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4a0e77fc847199f090a0457533a64 in term 1.
I20260806 02:42:57.137049 16255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "8d13cebf17284aefa5bc8ca736d9132c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4a0e77fc847199f090a0457533a64" is_pre_election: true
I20260806 02:42:57.137262 16150 leader_election.cc:302] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b4a0e77fc847199f090a0457533a64, f587dc5c1c0347ae9124929fa875a662; no voters:
I20260806 02:42:57.137291 16255 raft_consensus.cc:2379] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d13cebf17284aefa5bc8ca736d9132c in current term 1: Already voted for candidate 76b4a0e77fc847199f090a0457533a64 in this term.
I20260806 02:42:57.137750 16391 raft_consensus.cc:2781] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:42:57.137952 16391 raft_consensus.cc:686] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 LEADER]: Becoming Leader. State: Replica: 76b4a0e77fc847199f090a0457533a64, State: Running, Role: LEADER
I20260806 02:42:57.138316 16391 consensus_queue.cc:237] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } }
I20260806 02:42:57.140267 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.128944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57509 (local address 127.28.191.1:42867)
0806 02:42:57.129375 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:57.129379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:57.132465 (+ 3086us) server_negotiation.cc:407] Connection header received
0806 02:42:57.132670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:57.132674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:57.132719 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:57.132763 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:42:57.135632 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.136319 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.139022 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.139603 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.139715 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.139758 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:57.140081 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 02:42:57.140152 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":135,"threads_started":1}
I20260806 02:42:57.140748 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42867 (local address 127.28.191.2:57509)
0806 02:42:57.132387 (+ 3992us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.132403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:57.132547 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:57.135344 (+ 2797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:57.135347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:57.135368 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:42:57.135494 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.135514 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.137626 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.137629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.138871 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:57.138883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.139817 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.139821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:57.139922 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.139946 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:42:57.140552 (+ 606us) client_negotiation.cc:240] Negotiation successful
0806 02:42:57.140623 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3868,"thread_start_us":102,"threads_started":1}
I20260806 02:42:57.141422 15636 catalog_manager.cc:5495] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b4a0e77fc847199f090a0457533a64 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "76b4a0e77fc847199f090a0457533a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b4a0e77fc847199f090a0457533a64" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 } health_report { overall_health: UNKNOWN } } }
I20260806 02:42:57.142515 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc1d919804f4a87b29c7331a25f1ed1" candidate_uuid: "8d13cebf17284aefa5bc8ca736d9132c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f587dc5c1c0347ae9124929fa875a662" is_pre_election: true
I20260806 02:42:57.142684 15864 raft_consensus.cc:2379] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d13cebf17284aefa5bc8ca736d9132c in current term 1: Already voted for candidate 76b4a0e77fc847199f090a0457533a64 in this term.
I20260806 02:42:57.143522 15965 leader_election.cc:302] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d13cebf17284aefa5bc8ca736d9132c; no voters: 76b4a0e77fc847199f090a0457533a64, f587dc5c1c0347ae9124929fa875a662
I20260806 02:42:57.143795 16388 raft_consensus.cc:2726] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:42:57.219015 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.214326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.28.191.3:35621)
0806 02:42:57.214498 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:57.214502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:57.214518 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:42:57.214607 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:57.214610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:57.214673 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:57.214802 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:42:57.215223 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.215969 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.217242 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:57.217569 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:57.217674 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:57.218014 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:42:57.218578 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:42:57.218704 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:57.218786 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:42:57.218843 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:42:57.226424 15864 raft_consensus.cc:1261] T 4bc1d919804f4a87b29c7331a25f1ed1 P f587dc5c1c0347ae9124929fa875a662 [term 1 FOLLOWER]: Refusing update from remote peer 76b4a0e77fc847199f090a0457533a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:42:57.226470 16045 raft_consensus.cc:1261] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c [term 1 FOLLOWER]: Refusing update from remote peer 76b4a0e77fc847199f090a0457533a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:42:57.227115 16391 consensus_queue.cc:1035] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f587dc5c1c0347ae9124929fa875a662" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:42:57.227665 16410 consensus_queue.cc:1035] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d13cebf17284aefa5bc8ca736d9132c" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:42:57.239562 16420 mvcc.cc:205] Tried to move back new op lower bound from 7315391189910212608 to 7315391189561425920. Current Snapshot: MvccSnapshot[applied={T|T < 7315391189910212608}]
I20260806 02:42:57.246708 15636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:42:57.246860 15636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:42:57.249336 15636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baf304c102384d2d8bf6681c5317f8eb: Sending DeleteTablet for 3 replicas of tablet 4bc1d919804f4a87b29c7331a25f1ed1
I20260806 02:42:57.249423 16417 mvcc.cc:205] Tried to move back new op lower bound from 7315391189910212608 to 7315391189561425920. Current Snapshot: MvccSnapshot[applied={T|T < 7315391189910212608 or (T in {7315391189910212608})}]
I20260806 02:42:57.250553 16024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:57 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:43002
I20260806 02:42:57.250892 16215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:57 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:60540
I20260806 02:42:57.251011 15837 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc1d919804f4a87b29c7331a25f1ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:42:57 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34184
I20260806 02:42:57.251821 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15702
W20260806 02:42:57.269297 15618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:42867 (error 108)
I20260806 02:42:57.270377 16427 tablet_replica.cc:330] T 4bc1d919804f4a87b29c7331a25f1ed1 P 8d13cebf17284aefa5bc8ca736d9132c: stopping tablet replica
I20260806 02:42:57.270798 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15941
I20260806 02:42:57.275311 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:57.274962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:42867 (local address 127.0.0.1:34200)
0806 02:42:57.275150 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.275223 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260806 02:42:57.275609 15618 catalog_manager.cc:4584] TS f587dc5c1c0347ae9124929fa875a662 (127.28.191.1:42867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bc1d919804f4a87b29c7331a25f1ed1: Network error: Client connection negotiation failed: client connection to 127.28.191.1:42867: connect: Connection refused (error 111)
I20260806 02:42:57.276230 16428 tablet_replica.cc:330] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: stopping tablet replica
I20260806 02:42:57.276688 16428 raft_consensus.cc:2229] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:42:57.277271 16428 raft_consensus.cc:2258] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:42:57.281687 16428 ts_tablet_manager.cc:1905] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:42:57.284085 16372 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:42:57.283414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32791 (local address 127.0.0.1:43006)
0806 02:42:57.283577 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:57.283591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:57.283678 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:57.283970 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:32791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 02:42:57.284792 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 16120
I20260806 02:42:57.285224 16428 ts_tablet_manager.cc:1918] T 4bc1d919804f4a87b29c7331a25f1ed1 P 76b4a0e77fc847199f090a0457533a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:42:57.297111 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 15589
I20260806 02:42:57.310487 29436 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:42:57.656904 29436 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:42:57.657014 29436 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:42:58.001889 29436 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:42:58.002 UTC [14212] LOG: received smart shutdown request
2026-08-06 02:42:58.006 UTC [14212] LOG: background worker "logical replication launcher" (PID 14226) exited with exit code 1
2026-08-06 02:42:58.007 UTC [14217] LOG: shutting down
2026-08-06 02:42:58.149 UTC [14212] LOG: database system is shut down
2026-08-06T02:42:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49243 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:42:58 dist-test-slave-tfpf krb5kdc[16442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:42:58 dist-test-slave-tfpf krb5kdc[16442](info): set up 2 sockets
Aug 06 02:42:58 dist-test-slave-tfpf krb5kdc[16442](info): commencing operation
krb5kdc: starting...
W20260806 02:42:59.502216 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:42:59 dist-test-slave-tfpf krb5kdc[16442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:42:59.627731 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46187
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:42:59.737485 16462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:59.737746 16462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:59.737824 16462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:59.741637 16462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:42:59.741719 16462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:59.741767 16462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:59.741822 16462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:42:59.741859 16462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:42:59.746866 16462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:59.747885 16462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:59.748813 16462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:59.757476 16468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:59.757520 16462 server_base.cc:1034] running on GCE node
W20260806 02:42:59.757472 16467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:59.757472 16471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:59.758222 16462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:59.759202 16462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:59.760361 16462 hybrid_clock.cc:648] HybridClock initialized: now 1785984179760335 us; error 38 us; skew 500 ppm
Aug 06 02:42:59 dist-test-slave-tfpf krb5kdc[16442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:59.763087 16462 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:42:59.763861 16462 webserver.cc:460] Webserver started at http://127.28.191.62:46011/ using document root <none> and password file <none>
I20260806 02:42:59.764357 16462 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:59.764412 16462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:59.764566 16462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:42:59.766546 16462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "e9dcd3ba10184490b2d211ffcf5cb754"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "e078478abe9b070ab69f3a50455a0624"
server_key_iv: "0daba42db4ebb0b841697f2e025a80cf"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.767035 16462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "e9dcd3ba10184490b2d211ffcf5cb754"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "e078478abe9b070ab69f3a50455a0624"
server_key_iv: "0daba42db4ebb0b841697f2e025a80cf"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.770555 16462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:42:59.772781 16481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:59.774036 16462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:42:59.774174 16462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "e9dcd3ba10184490b2d211ffcf5cb754"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "e078478abe9b070ab69f3a50455a0624"
server_key_iv: "0daba42db4ebb0b841697f2e025a80cf"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.774278 16462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:42:59.793934 16462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:42:59.796826 16462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:42:59.797030 16462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:42:59.804136 16462 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:46187
I20260806 02:42:59.804154 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:46187 every 8 connection(s)
I20260806 02:42:59.805109 16462 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:42:59.807842 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:42:59.811605 16556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:42:59.814556 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 16462
I20260806 02:42:59.814714 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:42:59.815032 29436 external_mini_cluster.cc:1428] Setting key ca526da094b12d209cb5107a6f702c0e
Aug 06 02:42:59 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.61@KRBTEST.COM; defaulting to no policy
I20260806 02:42:59.826175 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.813741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46187 (local address 127.0.0.1:34618)
0806 02:42:59.814191 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.814232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:42:59.815202 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:42:59.815861 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:42:59.815866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:42:59.816303 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:42:59.816552 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:59.816567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:59.817799 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:59.817808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:59.818511 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:42:59.818521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:42:59.819175 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:42:59.819179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:42:59.819275 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:59.819922 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:42:59.819938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:42:59.821371 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:42:59.823226 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:59.823237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:59.823246 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:59.823467 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:59.823847 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:42:59.823850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:42:59.823852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:42:59.823962 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:42:59.824535 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:42:59.824541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:42:59.824677 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 02:42:59.824851 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 02:42:59.825193 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.61@KRBTEST.COM" created.
I20260806 02:42:59.826175 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.814552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.28.191.62:46187)
0806 02:42:59.814910 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 02:42:59.814921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:42:59.814965 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:42:59.815491 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:42:59.815497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:42:59.815676 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:42:59.815862 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:42:59.816699 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:59.817659 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:59.818630 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:42:59.818996 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:42:59.819182 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:42:59.821505 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:42:59.821518 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:42:59.821527 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:42:59.821547 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:42:59.823117 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:59.823605 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:59.823609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:59.823611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:59.823750 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:42:59.824066 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:42:59.824069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:42:59.824071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:42:59.824410 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:42:59.824560 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:42:59.824925 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 02:42:59.825248 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260806 02:42:59.828718 16556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:42:59.829527 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.829277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33662)
0806 02:42:59.829392 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.829468 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:42:59.829794 16484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
I20260806 02:42:59.830649 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.830443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33670)
0806 02:42:59.830555 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.830598 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:42:59.830888 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:43665: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
I20260806 02:42:59.842051 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:43665
--webserver_interface=127.28.191.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:42:59.880148 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } attempt: 1
I20260806 02:42:59.881093 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.880750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33678)
0806 02:42:59.880906 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.880969 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:42:59.881865 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.881641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33694)
0806 02:42:59.881753 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.881806 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:42:59.881994 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:43665: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
I20260806 02:42:59.950186 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } attempt: 2
I20260806 02:42:59.951004 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.950716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33708)
0806 02:42:59.950860 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.950929 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:42:59.951711 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:42:59.951500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33714)
0806 02:42:59.951608 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:42:59.951659 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:42:59.951838 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:43665: Network error: Client connection negotiation failed: client connection to 127.28.191.61:43665: connect: Connection refused (error 111)
W20260806 02:42:59.954241 16565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:42:59.954479 16565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:42:59.954564 16565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:42:59.958633 16565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:42:59.958709 16565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:42:59.958745 16565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:42:59.958779 16565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:42:59.958814 16565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:42:59.963972 16565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:43665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.191.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:42:59.964942 16565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:42:59.966050 16565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:42:59.974468 16572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:59.974493 16575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:42:59.974465 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:42:59.974913 16565 server_base.cc:1034] running on GCE node
I20260806 02:42:59.975445 16565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:42:59.976504 16565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:42:59.977710 16565 hybrid_clock.cc:648] HybridClock initialized: now 1785984179977689 us; error 55 us; skew 500 ppm
Aug 06 02:42:59 dist-test-slave-tfpf krb5kdc[16442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:42:59.980556 16565 init.cc:377] Logged in from keytab as kudu/127.28.191.61@KRBTEST.COM (short username kudu)
I20260806 02:42:59.981467 16565 webserver.cc:460] Webserver started at http://127.28.191.61:38993/ using document root <none> and password file <none>
I20260806 02:42:59.982021 16565 fs_manager.cc:359] Metadata directory not provided
I20260806 02:42:59.982118 16565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:42:59.982360 16565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:42:59.984356 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data/instance:
uuid: "d1241f2abe6e437c97a55195ec828d00"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "b43ebfe64d2b8937376509802790b52e"
server_key_iv: "87f132492a8ef243c2c2f0ca1d7dfa46"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.984928 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal/instance:
uuid: "d1241f2abe6e437c97a55195ec828d00"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "b43ebfe64d2b8937376509802790b52e"
server_key_iv: "87f132492a8ef243c2c2f0ca1d7dfa46"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.988593 16565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:42:59.991101 16585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:42:59.992340 16565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:42:59.992501 16565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "d1241f2abe6e437c97a55195ec828d00"
format_stamp: "Formatted at 2026-08-06 02:42:59 on dist-test-slave-tfpf"
server_key: "b43ebfe64d2b8937376509802790b52e"
server_key_iv: "87f132492a8ef243c2c2f0ca1d7dfa46"
server_key_version: "encryptionkey@0"
I20260806 02:42:59.992636 16565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:00.006814 16565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:00.010066 16565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:00.010284 16565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:00.017640 16565 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.61:43665
I20260806 02:43:00.017658 16660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.61:43665 every 8 connection(s)
I20260806 02:43:00.018630 16565 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
I20260806 02:43:00.021427 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:00.024998 16661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:43:00.028527 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 16565
I20260806 02:43:00.028656 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-1/wal/instance
I20260806 02:43:00.028921 29436 external_mini_cluster.cc:1428] Setting key 9e1495cc6701a31d1d4f23aa0dba9f04
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.60@KRBTEST.COM; defaulting to no policy
I20260806 02:43:00.038792 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.026929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.28.191.62:46187)
0806 02:43:00.027077 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.027081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.027764 (+ 683us) server_negotiation.cc:407] Connection header received
0806 02:43:00.028708 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.028711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.028773 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.028880 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.030052 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.030597 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.031744 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.031993 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.032116 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.034537 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.034540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.034548 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.034570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.036651 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.037227 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.037232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.037234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.037317 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.037667 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.037670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.037672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.037854 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.037964 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.038486 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.038620 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:43:00.039568 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.027182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46187 (local address 127.0.0.1:34626)
0806 02:43:00.027582 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.027622 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.028482 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.029131 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.029139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.029455 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.029903 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.029916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.030731 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.030734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.031611 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.031621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.032153 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.032157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.032319 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.032895 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.032912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.034367 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.036784 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.036790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.036803 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.037114 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.037411 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.037415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.037417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.037561 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.037963 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.037969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.038232 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.038412 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.038717 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.60@KRBTEST.COM" created.
I20260806 02:43:00.041189 16661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:43:00.042990 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } attempt: 3
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:43:00.054021 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.041721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33718)
0806 02:43:00.042587 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.042599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.042671 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.043131 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.043134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.043316 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.043423 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.043430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.044495 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.044502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.045570 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.045582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.046279 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.046282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.046370 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.046861 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.046872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.048596 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.052083 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.052086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.052087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.052401 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.052719 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.052721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.052723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.052795 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.053413 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.053415 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.053557 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.053683 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.053810 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796}
I20260806 02:43:00.054085 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.041804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.28.191.61:43665)
0806 02:43:00.042085 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.042094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.042679 (+ 585us) server_negotiation.cc:407] Connection header received
0806 02:43:00.042761 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.042770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.042986 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.043134 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.043538 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.044359 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.045701 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.046164 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.046285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.049271 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.049280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.049290 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.049311 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.051969 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.052520 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.052526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.052528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.052626 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.052913 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.052917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.052919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.053297 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.053466 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.053769 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.053895 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260806 02:43:00.054499 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.043372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33728)
0806 02:43:00.043540 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.043554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.043626 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.045481 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.045484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.045667 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.045778 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.045785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.046964 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.046971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.047706 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.047717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.048192 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.048196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.048301 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.048759 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.048772 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.050204 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.052601 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.052604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.052606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.052981 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.053393 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.053399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.053401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.053504 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.053966 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.053969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.054077 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.054199 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.054345 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 02:43:00.054641 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.043450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.28.191.61:43665)
0806 02:43:00.045278 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.045294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.045305 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:43:00.045332 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.045334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.045364 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.045463 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.045908 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.046816 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.047831 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.048079 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.048189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.050376 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.050381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.050385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.050416 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.052486 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.053170 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.053195 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.053197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.053277 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.053650 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.053653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.053655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.053858 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.053969 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.054272 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.054367 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1761,"spinlock_wait_cycles":3328,"thread_start_us":185,"threads_started":1}
I20260806 02:43:00.056437 16556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:43:00.056826 16661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:43:00.057240 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.056932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52876)
0806 02:43:00.057119 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.057176 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:43:00.057641 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.057282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52884)
0806 02:43:00.057512 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.057589 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260806 02:43:00.057834 16588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.057979 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.057747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52898)
0806 02:43:00.057874 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.057928 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 02:43:00.058117 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.058787 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.058558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52906)
0806 02:43:00.058663 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.058740 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
W20260806 02:43:00.058957 16661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.059912 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:33199
--webserver_interface=127.28.191.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:00.089268 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 1
I20260806 02:43:00.089980 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.089721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52916)
0806 02:43:00.089850 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.089909 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:43:00.090667 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.090464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52932)
0806 02:43:00.090564 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.090609 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:43:00.090790 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.108206 16661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 1
I20260806 02:43:00.109092 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.108739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52944)
0806 02:43:00.108905 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.108973 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:43:00.109802 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.109584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52958)
0806 02:43:00.109693 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.109743 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:43:00.109930 16661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.165938 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 2
I20260806 02:43:00.166813 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.166491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52972)
0806 02:43:00.166666 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.166735 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:43:00.167567 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.167354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52976)
0806 02:43:00.167461 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.167514 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:43:00.167798 16556 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
W20260806 02:43:00.172900 16672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:00.173153 16672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:00.173259 16672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:00.177206 16672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:00.177254 16672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:00.177274 16672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:00.177291 16672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:00.177306 16672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:00.178078 16661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 2
I20260806 02:43:00.178889 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.178583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52982)
0806 02:43:00.178747 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.178810 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:43:00.179590 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.179375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52988)
0806 02:43:00.179479 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.179529 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:43:00.179730 16661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:33199: Network error: Client connection negotiation failed: client connection to 127.28.191.60:33199: connect: Connection refused (error 111)
I20260806 02:43:00.183055 16672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46187,127.28.191.61:43665,127.28.191.60:33199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.191.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:00.184062 16672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:00.185105 16672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:00.193455 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:00.193522 16679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:00.193571 16672 server_base.cc:1034] running on GCE node
W20260806 02:43:00.193455 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:00.194408 16672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:00.195694 16672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:00.196921 16672 hybrid_clock.cc:648] HybridClock initialized: now 1785984180196886 us; error 37 us; skew 500 ppm
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:00.199587 16672 init.cc:377] Logged in from keytab as kudu/127.28.191.60@KRBTEST.COM (short username kudu)
I20260806 02:43:00.200423 16672 webserver.cc:460] Webserver started at http://127.28.191.60:38245/ using document root <none> and password file <none>
I20260806 02:43:00.200976 16672 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:00.201067 16672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:00.201323 16672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:00.203290 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data/instance:
uuid: "84d71106dc884cc5a9dca8b9a00f61e2"
format_stamp: "Formatted at 2026-08-06 02:43:00 on dist-test-slave-tfpf"
server_key: "8711c8556365c5b6323d4572af6b21a6"
server_key_iv: "363d25eaabc90ddf07a1cb54272b3ac1"
server_key_version: "encryptionkey@0"
I20260806 02:43:00.203855 16672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal/instance:
uuid: "84d71106dc884cc5a9dca8b9a00f61e2"
format_stamp: "Formatted at 2026-08-06 02:43:00 on dist-test-slave-tfpf"
server_key: "8711c8556365c5b6323d4572af6b21a6"
server_key_iv: "363d25eaabc90ddf07a1cb54272b3ac1"
server_key_version: "encryptionkey@0"
I20260806 02:43:00.207476 16672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 02:43:00.209861 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:00.211001 16672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:43:00.211143 16672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "84d71106dc884cc5a9dca8b9a00f61e2"
format_stamp: "Formatted at 2026-08-06 02:43:00 on dist-test-slave-tfpf"
server_key: "8711c8556365c5b6323d4572af6b21a6"
server_key_iv: "363d25eaabc90ddf07a1cb54272b3ac1"
server_key_version: "encryptionkey@0"
I20260806 02:43:00.211253 16672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:00.225369 16672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:00.228123 16672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:00.228320 16672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:00.235170 16672 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.60:33199
I20260806 02:43:00.235232 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.60:33199 every 8 connection(s)
I20260806 02:43:00.236117 16672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
I20260806 02:43:00.237587 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 16672
I20260806 02:43:00.237713 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984016215184-29436-0/minicluster-data/master-2/wal/instance
I20260806 02:43:00.238023 29436 external_mini_cluster.cc:1428] Setting key ad3be27f494fef9c18176f5885410b8c
I20260806 02:43:00.239365 16772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:00.243949 16772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:00.253340 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.239372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.28.191.60:33199)
0806 02:43:00.239804 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.239823 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.239864 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:43:00.240658 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.240680 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.241093 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.241437 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.242065 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.243556 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.244408 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.245095 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.245261 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.248127 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.248131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.248140 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.248166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.250116 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.250558 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.250565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.250570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.250692 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.251024 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.251027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.251028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.251415 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.251610 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.251949 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.252244 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260806 02:43:00.255365 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.245798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46187 (local address 127.0.0.1:34632)
0806 02:43:00.246185 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.246233 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.246350 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.246673 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.246677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.247138 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.247320 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.247328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.248297 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.248303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.249319 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.249333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.249768 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.249772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.249896 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.250401 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.250413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.251804 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.253681 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.253689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.253693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.253905 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.254202 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.254204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.254206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.254302 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.254723 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.254729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.254841 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.254985 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.255190 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":162,"threads_started":1}
I20260806 02:43:00.255373 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.245873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.28.191.62:46187)
0806 02:43:00.246018 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.246022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.246318 (+ 296us) server_negotiation.cc:407] Connection header received
0806 02:43:00.246453 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.246458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.246516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.246619 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.247457 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.248182 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.249442 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.249654 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.249735 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.251938 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.251942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.251945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.251970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.253576 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.254006 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.254009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.254011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.254109 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.254409 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.254412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.254413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.254619 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.254698 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.255114 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.255239 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:43:00.256796 16772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:43:00.265378 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.255726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.28.191.62:46187)
0806 02:43:00.255824 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.255828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.255938 (+ 110us) server_negotiation.cc:407] Connection header received
0806 02:43:00.256079 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.256082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.256126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.256204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.256990 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.257734 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.258689 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.259001 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.259123 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.261874 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.261878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.261881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.261906 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.263585 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264095 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.264098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.264099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.264171 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264470 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.264474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.264476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.264656 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.264910 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.265020 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.265181 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 02:43:00.265839 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.257365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:43665 (local address 127.0.0.1:33734)
0806 02:43:00.257596 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.257607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.257669 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.257905 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.257908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.258101 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.258211 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.258218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.259164 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.259167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.259928 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.259941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.260401 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.260405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.260524 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.260980 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.260991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.262359 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.264255 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.264260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.264450 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.264746 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.264751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.264835 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.265346 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.265349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.265446 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.265607 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.265727 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260806 02:43:00.265974 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.257485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.28.191.61:43665)
0806 02:43:00.257618 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.257623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.257641 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:43:00.257735 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.257739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.257795 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.257873 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.258330 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.259003 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.260055 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.260292 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.260374 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.262484 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.262487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.262489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.262511 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.264147 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264565 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.264573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.264574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.264646 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.264923 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.264927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.264929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.265192 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.265299 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.265696 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.265830 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:43:00.266749 16772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:43:00.266950 16556 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 3
I20260806 02:43:00.270872 16661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } attempt: 3
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:43:00.277179 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.267306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:53000)
0806 02:43:00.267467 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.267481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.267572 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.267985 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.267988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.268172 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.268257 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.268263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.269736 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.269739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.270327 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.270335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.270840 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.270844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.270930 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.271380 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.271395 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.272969 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.275434 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.275437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.275439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.275671 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.276005 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.276010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.276013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.276119 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.276577 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.276583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.276687 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.276796 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.276941 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:43:00.277842 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.267447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.28.191.60:33199)
0806 02:43:00.267777 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.267786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.267796 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:43:00.267824 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.267826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.267875 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.267966 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.268366 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.269628 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.270419 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.270719 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.270840 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.273142 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.273148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.273154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.273185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.275323 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.275800 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.275805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.275808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.275901 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.276232 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.276236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.276238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.276459 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.277410 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.277493 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.277619 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":264,"spinlock_wait_cycles":3584,"thread_start_us":118,"threads_started":1}
Aug 06 02:43:00 dist-test-slave-tfpf krb5kdc[16442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984179, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:43:00.278638 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.267235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:52996)
0806 02:43:00.267341 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.267355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.267705 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.268398 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.268400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.268546 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.268622 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.268628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.270475 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.270479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.271756 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.271771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.272404 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.272411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.272500 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.273261 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.273273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.274734 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.277112 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.277115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.277118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.277361 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.277677 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.277680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.277682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.277765 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.278161 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.278167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.278256 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.278333 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.278445 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":368}
I20260806 02:43:00.279217 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.267447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.28.191.60:33199)
0806 02:43:00.267523 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.267529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.267542 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:43:00.267818 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.267821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.267874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.267950 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.268741 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.269628 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.271890 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.272269 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.272369 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.274874 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.274880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.274883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.274910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.276966 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.277484 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.277487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.277489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.277582 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.277869 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.277872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.277874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.278053 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.278815 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.278880 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.279012 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":24,"spinlock_wait_cycles":2432}
I20260806 02:43:00.281612 16556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: Bootstrap starting.
I20260806 02:43:00.282250 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.271499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.28.191.60:33199)
0806 02:43:00.272547 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.272558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.272568 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:43:00.272599 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.272602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.272644 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.272736 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.273212 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.273998 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.275080 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.276189 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.278591 (+ 2402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.278889 (+ 298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.278893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.278896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.278914 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.280642 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.281163 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.281166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.281168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.281263 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.281558 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.281560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.281561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.281704 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.281817 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.282049 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.282137 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":105,"threads_started":1}
I20260806 02:43:00.282286 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.271251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:33199 (local address 127.0.0.1:53014)
0806 02:43:00.271523 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:00.271548 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:00.271622 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:00.272776 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:00.272781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:00.272969 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:00.273109 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.273117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.274090 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.274096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.274755 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:00.274764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.276323 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.276329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:00.276429 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.276891 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:00.276907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:00.278463 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:00.280755 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.280758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.280760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.281014 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.281351 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:00.281356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:00.281361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:00.281459 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:00.281807 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:00.281809 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:00.281901 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:43:00.282009 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:43:00.282144 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260806 02:43:00.284224 16772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: Bootstrap starting.
I20260806 02:43:00.284519 16661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: Bootstrap starting.
I20260806 02:43:00.286582 16661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:00.286721 16772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:00.286746 16556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:00.287482 16661 log.cc:824] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:00.287554 16556 log.cc:824] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:00.287561 16772 log.cc:824] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:00.287637 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.266971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.28.191.61:43665)
0806 02:43:00.267089 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.267095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.267107 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:43:00.267138 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.267144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.267185 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.267256 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.268088 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.268771 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.269562 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.269845 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.269943 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.273612 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.273615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.273618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.273641 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.276383 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.276905 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.276909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.276911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.277031 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.277747 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.277751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.277753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.286826 (+ 9073us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.286974 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.287379 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.287482 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:43:00.289569 16772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: No bootstrap required, opened a new log
I20260806 02:43:00.289594 16556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: No bootstrap required, opened a new log
I20260806 02:43:00.289594 16661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: No bootstrap required, opened a new log
I20260806 02:43:00.292834 16772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.293059 16772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:00.293408 16556 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.293408 16661 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.293529 16772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d71106dc884cc5a9dca8b9a00f61e2, State: Initialized, Role: FOLLOWER
I20260806 02:43:00.293609 16661 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:00.293641 16556 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:00.293696 16661 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1241f2abe6e437c97a55195ec828d00, State: Initialized, Role: FOLLOWER
I20260806 02:43:00.293712 16556 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dcd3ba10184490b2d211ffcf5cb754, State: Initialized, Role: FOLLOWER
I20260806 02:43:00.294011 16772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.294123 16661 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.294123 16556 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.295449 16661 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:00.295449 16556 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:00.295450 16772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:00.295797 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 02:43:00.298625 16488 tablet.cc:2369] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:43:00.301133 16845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:43:00.301347 16841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:43:00.301425 16841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:43:00.302150 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.302306 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:43:00.304203 16846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:43:00.304558 16846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:43:00.302547 16786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.306037 16786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:43:00.306175 16845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:43:00.307479 16462 master_runner.cc:263] No leader master found from master e9dcd3ba10184490b2d211ffcf5cb754
I20260806 02:43:00.307539 16462 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:43:00.309523 16565 master_runner.cc:263] No leader master found from master d1241f2abe6e437c97a55195ec828d00
I20260806 02:43:00.309603 16565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:43:00.309780 16787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.309916 16787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:43:00.310249 16672 master_runner.cc:263] No leader master found from master 84d71106dc884cc5a9dca8b9a00f61e2
I20260806 02:43:00.310321 16672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:43:00.314565 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.301970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.28.191.61:43665)
0806 02:43:00.302093 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.302097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.304525 (+ 2428us) server_negotiation.cc:407] Connection header received
0806 02:43:00.304860 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.304865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.304908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.304956 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.306346 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.307031 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.308114 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.308428 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.308503 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.310922 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.310926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.310928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.310954 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.312622 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.313161 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.313165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.313168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.313257 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.313566 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.313569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.313571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.313792 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.313916 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.314287 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.314411 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:43:00.314805 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.298523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.28.191.62:46187)
0806 02:43:00.298632 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.298636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.303026 (+ 4390us) server_negotiation.cc:407] Connection header received
0806 02:43:00.303230 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.303234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.303279 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.303331 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.304364 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.305137 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.306345 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.306694 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.306773 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.311093 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.311097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.311099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.311126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.312885 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.313395 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.313399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.313400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.313485 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.313828 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.313834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.313836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.314066 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.314173 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.314505 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.314642 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:43:00.322777 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:00.302538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.28.191.60:33199)
0806 02:43:00.302649 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:00.302653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:00.314193 (+ 11540us) server_negotiation.cc:407] Connection header received
0806 02:43:00.314400 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:00.314404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:00.314448 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:00.314496 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:00.315519 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.316177 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.317114 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:00.317452 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:00.317550 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:00.319195 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:00.319199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:00.319201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:00.319229 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:00.320908 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.321435 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.321439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.321441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.321528 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:00.321836 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:00.321839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:00.321841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:00.322036 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:00.322157 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:00.322505 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 02:43:00.322622 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 02:43:00.434489 16787 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:00.434698 16787 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.435745 16787 leader_election.cc:288] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1241f2abe6e437c97a55195ec828d00 (127.28.191.61:43665), 84d71106dc884cc5a9dca8b9a00f61e2 (127.28.191.60:33199)
I20260806 02:43:00.436574 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" is_pre_election: true
I20260806 02:43:00.437232 16737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9dcd3ba10184490b2d211ffcf5cb754 in term 0.
I20260806 02:43:00.437762 16484 leader_election.cc:302] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d71106dc884cc5a9dca8b9a00f61e2, e9dcd3ba10184490b2d211ffcf5cb754; no voters:
I20260806 02:43:00.438156 16787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:43:00.438247 16787 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:43:00.438297 16787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:00.439734 16787 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.440074 16787 leader_election.cc:288] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [CANDIDATE]: Term 1 election: Requested vote from peers d1241f2abe6e437c97a55195ec828d00 (127.28.191.61:43665), 84d71106dc884cc5a9dca8b9a00f61e2 (127.28.191.60:33199)
I20260806 02:43:00.440610 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d71106dc884cc5a9dca8b9a00f61e2"
I20260806 02:43:00.440726 16737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:00.442015 16737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9dcd3ba10184490b2d211ffcf5cb754 in term 1.
I20260806 02:43:00.442396 16484 leader_election.cc:302] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d71106dc884cc5a9dca8b9a00f61e2, e9dcd3ba10184490b2d211ffcf5cb754; no voters:
I20260806 02:43:00.443111 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1241f2abe6e437c97a55195ec828d00" is_pre_election: true
I20260806 02:43:00.443394 16619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9dcd3ba10184490b2d211ffcf5cb754 in term 0.
I20260806 02:43:00.443671 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1241f2abe6e437c97a55195ec828d00"
I20260806 02:43:00.443771 16621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:00.444933 16621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9dcd3ba10184490b2d211ffcf5cb754 in term 1.
I20260806 02:43:00.445641 16787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:00.445926 16787 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [term 1 LEADER]: Becoming Leader. State: Replica: e9dcd3ba10184490b2d211ffcf5cb754, State: Running, Role: LEADER
I20260806 02:43:00.446278 16787 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } }
I20260806 02:43:00.447927 16787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dcd3ba10184490b2d211ffcf5cb754. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.448101 16787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:00.450887 16861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:00.451833 16861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:00.458830 16737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [term 1 FOLLOWER]: Refusing update from remote peer e9dcd3ba10184490b2d211ffcf5cb754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:00.459640 16858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:00.464154 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dcd3ba10184490b2d211ffcf5cb754. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.464287 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:43:00.467504 16621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [term 1 FOLLOWER]: Refusing update from remote peer e9dcd3ba10184490b2d211ffcf5cb754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:00.468096 16858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:00.471730 16786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dcd3ba10184490b2d211ffcf5cb754. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.471843 16786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:43:00.475585 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.475711 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:00.476821 16786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.476946 16786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:43:00.477135 16861 catalog_manager.cc:1347] Generated new cluster ID: 39df4c99dfa1414bb433a3e95d7508f8
I20260806 02:43:00.477200 16861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260806 02:43:00.479703 16698 tablet.cc:2369] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:43:00.485177 16862 mvcc.cc:205] Tried to move back new op lower bound from 7315391203151200256 to 7315391203110952960. Current Snapshot: MvccSnapshot[applied={T|T < 7315391203151200256 or (T in {7315391203151200256})}]
I20260806 02:43:00.486912 16866 mvcc.cc:205] Tried to move back new op lower bound from 7315391203151200256 to 7315391203110952960. Current Snapshot: MvccSnapshot[applied={T|T < 7315391203151200256}]
I20260806 02:43:00.487571 16787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.487684 16787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:00.488623 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9dcd3ba10184490b2d211ffcf5cb754" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46187 } } peers { permanent_uuid: "d1241f2abe6e437c97a55195ec828d00" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 43665 } } peers { permanent_uuid: "84d71106dc884cc5a9dca8b9a00f61e2" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 33199 } } }
I20260806 02:43:00.488765 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:43:00.489283 16863 mvcc.cc:205] Tried to move back new op lower bound from 7315391203151200256 to 7315391203110952960. Current Snapshot: MvccSnapshot[applied={T|T < 7315391203151200256}]
I20260806 02:43:00.495929 16861 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:43:00.497696 16861 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:00.507409 16861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9dcd3ba10184490b2d211ffcf5cb754: Generated new TSK 0
I20260806 02:43:00.508255 16861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:43:01.302744 16841 catalog_manager.cc:1259] Loaded cluster ID: 39df4c99dfa1414bb433a3e95d7508f8
I20260806 02:43:01.302834 16841 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: loading cluster ID for follower catalog manager: success
I20260806 02:43:01.304533 16841 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: acquiring CA information for follower catalog manager: success
I20260806 02:43:01.305799 16841 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84d71106dc884cc5a9dca8b9a00f61e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:43:01.307258 16845 catalog_manager.cc:1259] Loaded cluster ID: 39df4c99dfa1414bb433a3e95d7508f8
I20260806 02:43:01.307318 16845 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: loading cluster ID for follower catalog manager: success
I20260806 02:43:01.308808 16845 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: acquiring CA information for follower catalog manager: success
I20260806 02:43:01.309842 16845 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1241f2abe6e437c97a55195ec828d00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:43:01.574679 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 16462
I20260806 02:43:01.589555 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 16565
I20260806 02:43:01.600512 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 16672
2026-08-06T02:43:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3329 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:43:01 dist-test-slave-tfpf krb5kdc[16877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:43:01 dist-test-slave-tfpf krb5kdc[16877](info): set up 2 sockets
Aug 06 02:43:01 dist-test-slave-tfpf krb5kdc[16877](info): commencing operation
krb5kdc: starting...
W20260806 02:43:02.822429 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:43:02 dist-test-slave-tfpf krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:43:02.946219 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42655
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:03.060835 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:03.061163 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:03.061269 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:03.065341 16896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:03.065423 16896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:03.065459 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:03.065497 16896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:03.065531 16896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:03.070624 16896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:03.071581 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:03.072574 16896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:03.080966 16903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:03.080996 16902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:03.081219 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:03.081260 16896 server_base.cc:1034] running on GCE node
I20260806 02:43:03.081897 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:03.082927 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:03.084117 16896 hybrid_clock.cc:648] HybridClock initialized: now 1785984183084104 us; error 26 us; skew 500 ppm
Aug 06 02:43:03 dist-test-slave-tfpf krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984183, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:03.086941 16896 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:03.087852 16896 webserver.cc:460] Webserver started at http://127.28.191.62:42517/ using document root <none> and password file <none>
I20260806 02:43:03.088436 16896 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:03.088551 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:03.088795 16896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:03.090862 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "891ab58e32e140d09e2c2ff48bb5f2b1"
format_stamp: "Formatted at 2026-08-06 02:43:03 on dist-test-slave-tfpf"
server_key: "de67bd4277b910c4cb45d6eb09b20b1b"
server_key_iv: "111b188ff0b9f59b5a934b8f12728ec6"
server_key_version: "encryptionkey@0"
I20260806 02:43:03.091429 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "891ab58e32e140d09e2c2ff48bb5f2b1"
format_stamp: "Formatted at 2026-08-06 02:43:03 on dist-test-slave-tfpf"
server_key: "de67bd4277b910c4cb45d6eb09b20b1b"
server_key_iv: "111b188ff0b9f59b5a934b8f12728ec6"
server_key_version: "encryptionkey@0"
I20260806 02:43:03.095615 16896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:43:03.098214 16915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:03.099586 16896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:43:03.099750 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "891ab58e32e140d09e2c2ff48bb5f2b1"
format_stamp: "Formatted at 2026-08-06 02:43:03 on dist-test-slave-tfpf"
server_key: "de67bd4277b910c4cb45d6eb09b20b1b"
server_key_iv: "111b188ff0b9f59b5a934b8f12728ec6"
server_key_version: "encryptionkey@0"
I20260806 02:43:03.099870 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:03.118652 16896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:03.121565 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:03.121768 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:03.129225 16896 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:42655
I20260806 02:43:03.129241 16987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:42655 every 8 connection(s)
I20260806 02:43:03.130208 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:03.132284 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 16896
I20260806 02:43:03.132424 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:43:03.132738 29436 external_mini_cluster.cc:1428] Setting key f44d97685d933aeee16ffcc123982131
I20260806 02:43:03.133168 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:03.139889 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1: Bootstrap starting.
Aug 06 02:43:03 dist-test-slave-tfpf krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:03.142400 16989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:03.143079 16989 log.cc:824] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:03.144774 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1: No bootstrap required, opened a new log
I20260806 02:43:03.147189 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:03.134126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.28.191.62:42655)
0806 02:43:03.134534 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:03.134552 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:03.134599 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:43:03.135212 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:03.135227 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:03.135528 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:03.135793 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:03.136742 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:03.137782 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:03.138951 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:03.139388 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:03.139498 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:03.142254 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:03.142259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:03.142271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:03.142296 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:03.144084 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:03.144527 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:03.144532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:03.144544 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:03.144653 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:03.144954 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:03.144958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:03.144959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:03.145468 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:03.145691 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:03.145934 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 02:43:03.146234 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":148,"threads_started":1}
I20260806 02:43:03.148751 16989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } }
I20260806 02:43:03.148980 16989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:03.149109 16989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891ab58e32e140d09e2c2ff48bb5f2b1, State: Initialized, Role: FOLLOWER
I20260806 02:43:03.149523 16989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } }
I20260806 02:43:03.149648 16989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:03.149729 16989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:03.149837 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:03.149986 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:43:03.150662 16989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } }
I20260806 02:43:03.151014 16989 leader_election.cc:302] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891ab58e32e140d09e2c2ff48bb5f2b1; no voters:
I20260806 02:43:03.151306 16989 leader_election.cc:288] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:43:03.151436 16996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:03.151713 16996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [term 1 LEADER]: Becoming Leader. State: Replica: 891ab58e32e140d09e2c2ff48bb5f2b1, State: Running, Role: LEADER
I20260806 02:43:03.152038 16996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } }
I20260806 02:43:03.152453 16989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:03.153553 17000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891ab58e32e140d09e2c2ff48bb5f2b1. Latest consensus state: current_term: 1 leader_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } } }
I20260806 02:43:03.153674 17000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:03.153960 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891ab58e32e140d09e2c2ff48bb5f2b1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42655 } } }
I20260806 02:43:03.154026 17012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:03.154806 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:03.156985 17012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:03.161975 17012 catalog_manager.cc:1347] Generated new cluster ID: a818016273e54751998bd7a7c4dc8679
I20260806 02:43:03.162045 17012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:43:03.171023 17012 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:43:03.171824 17012 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:03.177597 17012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891ab58e32e140d09e2c2ff48bb5f2b1: Generated new TSK 0
I20260806 02:43:03.178160 17012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:43:03.205933 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 16896
2026-08-06T02:43:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1609 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:43:03 dist-test-slave-tfpf krb5kdc[17031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:43:03 dist-test-slave-tfpf krb5kdc[17031](info): set up 2 sockets
Aug 06 02:43:03 dist-test-slave-tfpf krb5kdc[17031](info): commencing operation
krb5kdc: starting...
W20260806 02:43:04.415469 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:43:04 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:43:04.535418 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:04.649091 17051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:04.649356 17051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:04.649423 17051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:04.653541 17051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:04.653592 17051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:04.653612 17051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:04.653628 17051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:04.653694 17051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:04.658794 17051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:04.659754 17051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:04.660754 17051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:04.669679 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:04.669741 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:04.669764 17051 server_base.cc:1034] running on GCE node
W20260806 02:43:04.669847 17057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:04.670504 17051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:04.671589 17051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:04.672785 17051 hybrid_clock.cc:648] HybridClock initialized: now 1785984184672757 us; error 43 us; skew 500 ppm
Aug 06 02:43:04 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984184, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:04.678779 17051 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:04.679714 17051 webserver.cc:460] Webserver started at http://127.28.191.62:38549/ using document root <none> and password file <none>
I20260806 02:43:04.680222 17051 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:04.680280 17051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:04.680444 17051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:04.682518 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "20969d1df7d64233b1981b8bdfcc6a70"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9648528a23482b433dba3c31bdaef089"
server_key_iv: "6ce9e71bcde1faff270fe2027fb4b774"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.683050 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "20969d1df7d64233b1981b8bdfcc6a70"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9648528a23482b433dba3c31bdaef089"
server_key_iv: "6ce9e71bcde1faff270fe2027fb4b774"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.686710 17051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:04.688973 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:04.690214 17051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:43:04.690343 17051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "20969d1df7d64233b1981b8bdfcc6a70"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9648528a23482b433dba3c31bdaef089"
server_key_iv: "6ce9e71bcde1faff270fe2027fb4b774"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.690444 17051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:04.705559 17051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:04.708667 17051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:04.708873 17051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:04.716058 17051 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:45297
I20260806 02:43:04.716334 17144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:45297 every 8 connection(s)
I20260806 02:43:04.717027 17051 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:04.719866 17145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:04.721939 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17051
I20260806 02:43:04.722088 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:43:04.722412 29436 external_mini_cluster.cc:1428] Setting key bc6278a0096201691790161b9784daa3
I20260806 02:43:04.726119 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap starting.
I20260806 02:43:04.728489 17145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:04.729357 17145 log.cc:824] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:43:04 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:04.731945 17145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: No bootstrap required, opened a new log
I20260806 02:43:04.736814 17145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:04.737190 17145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:04.737250 17145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Initialized, Role: FOLLOWER
I20260806 02:43:04.737766 17145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:04.737949 17145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:04.738030 17145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:04.738128 17145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:04.739306 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:04.723814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.28.191.62:45297)
0806 02:43:04.724249 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:04.724267 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:04.724305 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:43:04.725176 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:04.725195 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:04.725522 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:04.725801 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:04.726750 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:04.727729 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:04.728856 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:04.729407 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:04.729628 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:04.732923 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:04.732931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:04.732946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:04.732979 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:04.735950 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:04.736416 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:04.736422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:04.736428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:04.736582 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:04.736881 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:04.736884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:04.736886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:04.737463 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:04.737676 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:04.737938 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:43:04.738275 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":160,"threads_started":1}
I20260806 02:43:04.739274 17145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:04.739748 17145 leader_election.cc:302] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20969d1df7d64233b1981b8bdfcc6a70; no voters:
I20260806 02:43:04.740105 17145 leader_election.cc:288] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:43:04.740201 17152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:04.740454 17152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 LEADER]: Becoming Leader. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Running, Role: LEADER
I20260806 02:43:04.740764 17152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:04.741971 17154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20969d1df7d64233b1981b8bdfcc6a70. Latest consensus state: current_term: 1 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:04.742192 17154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:04.742422 17153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:04.742527 17153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:04.742580 17145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:04.742779 17158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:04.745924 17158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:04.752350 17158 catalog_manager.cc:1347] Generated new cluster ID: ff1ac29959d347cf955d5284c9348b61
I20260806 02:43:04.752430 17158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:43:04.787674 17158 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:43:04.788970 17158 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:04.798770 17158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Generated new TSK 0
I20260806 02:43:04.799515 17158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:43:04.837884 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:04.955049 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:04.955309 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:04.955349 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:04.959218 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:04.959266 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:04.959321 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:43:04.964330 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:04.965190 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:04.966459 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:04.978400 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:04.978400 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:04.978425 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:04.978593 17187 server_base.cc:1034] running on GCE node
I20260806 02:43:04.979310 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:04.979946 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:04.981130 17187 hybrid_clock.cc:648] HybridClock initialized: now 1785984184981115 us; error 30 us; skew 500 ppm
Aug 06 02:43:04 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984184, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:04.983927 17187 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:43:04.984778 17187 webserver.cc:460] Webserver started at http://127.28.191.1:35623/ using document root <none> and password file <none>
I20260806 02:43:04.985409 17187 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:04.985502 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:04.985747 17187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:04.987785 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9677ec1ebae9fa51c71262cac9071557"
server_key_iv: "08480c7e6b07595d6d4d40f3fdc9c4ef"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.988385 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9677ec1ebae9fa51c71262cac9071557"
server_key_iv: "08480c7e6b07595d6d4d40f3fdc9c4ef"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.992250 17187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:43:04.994872 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:04.996106 17187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:04.996256 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9677ec1ebae9fa51c71262cac9071557"
server_key_iv: "08480c7e6b07595d6d4d40f3fdc9c4ef"
server_key_version: "encryptionkey@0"
I20260806 02:43:04.996374 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:05.017601 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:05.020972 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:05.021260 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:05.021876 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:05.022750 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:05.022821 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.022891 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:05.022941 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.031656 17187 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:40593
I20260806 02:43:05.031734 17360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:40593 every 8 connection(s)
I20260806 02:43:05.032567 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:43:05.035359 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17187
I20260806 02:43:05.035495 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:43:05.035825 29436 external_mini_cluster.cc:1428] Setting key bc5dc63490c3d07bed3848e0e32d3f7d
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984184, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:43:05.048066 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.034369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:38371 (local address 127.28.191.62:45297)
0806 02:43:05.034499 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.034503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.035364 (+ 861us) server_negotiation.cc:407] Connection header received
0806 02:43:05.036995 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.036998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.037040 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.037174 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:05.038387 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.039229 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.041113 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.041375 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.041467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.044267 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:05.044270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:05.044272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:05.044295 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:05.045914 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.046422 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.046427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.046429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.046515 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.046870 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.046875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.046877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.047061 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:05.047175 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.047756 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.047888 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:43:05.048915 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.034767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:38371)
0806 02:43:05.035210 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.035252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.036785 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.037355 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.037365 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.037735 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:05.038212 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.038225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.039389 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.039395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.040867 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.040882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.041517 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.041520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.041701 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.042305 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:05.042321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:05.044076 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:05.046036 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.046042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.046056 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.046317 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.046625 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.046628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.046630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.046771 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.047186 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:05.047194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:05.047433 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.047711 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.048027 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:43:05.050110 17361 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:05.050367 17361 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:05.050839 17361 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
I20260806 02:43:05.052496 17087 ts_manager.cc:194] Registered new tserver with Master: 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593)
I20260806 02:43:05.053766 17087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:38371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:43:05.066007 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:05.178610 17371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:05.178887 17371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:05.178985 17371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:05.183022 17371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:05.183123 17371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:05.183243 17371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:43:05.188525 17371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:05.189674 17371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:05.190713 17371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:05.200943 17381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.200986 17380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.200973 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:05.201422 17371 server_base.cc:1034] running on GCE node
I20260806 02:43:05.201849 17371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:05.202477 17371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:05.203672 17371 hybrid_clock.cc:648] HybridClock initialized: now 1785984185203643 us; error 51 us; skew 500 ppm
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:05.206496 17371 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:43:05.207345 17371 webserver.cc:460] Webserver started at http://127.28.191.2:40597/ using document root <none> and password file <none>
I20260806 02:43:05.207911 17371 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:05.208003 17371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:05.208240 17371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:05.210297 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "de60135ccf99aff78a444d73fea73c45"
server_key_iv: "a975fa87570c5b765d1b0db28465052b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.210875 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "de60135ccf99aff78a444d73fea73c45"
server_key_iv: "a975fa87570c5b765d1b0db28465052b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.214530 17371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:43:05.216828 17394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.218067 17371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:05.218186 17371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "de60135ccf99aff78a444d73fea73c45"
server_key_iv: "a975fa87570c5b765d1b0db28465052b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.218286 17371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:05.251801 17371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:05.255200 17371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:05.255419 17371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:05.256044 17371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:05.256904 17371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:05.256955 17371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.256995 17371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:05.257014 17371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.266279 17371 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:32909
I20260806 02:43:05.266309 17571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:32909 every 8 connection(s)
I20260806 02:43:05.267235 17371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:43:05.272943 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17371
I20260806 02:43:05.273168 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:43:05.273526 29436 external_mini_cluster.cc:1428] Setting key f44a3976e5b385dda06e6759d48d166f
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:05.281644 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.268987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:60425 (local address 127.28.191.62:45297)
0806 02:43:05.269176 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.269180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.269950 (+ 770us) server_negotiation.cc:407] Connection header received
0806 02:43:05.271270 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.271273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.271320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.271395 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:05.272480 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.273116 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.274435 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.274677 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.274789 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.277690 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:05.277695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:05.277697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:05.277724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:05.279264 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.279880 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.279883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.279884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.279961 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.280371 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.280373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.280374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.280569 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:05.280645 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.281385 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.281513 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:43:05.282776 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.269359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:60425)
0806 02:43:05.269792 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.269834 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.271064 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.271535 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.271541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.271886 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:05.272341 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.272354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.273251 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.273273 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.274298 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.274311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.274825 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.274829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.275002 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.275605 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:05.275622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:05.277488 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:05.279380 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.279389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.279407 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.279784 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.280077 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.280083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.280088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.280277 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.280676 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:05.280686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:05.281044 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.281302 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.281655 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260806 02:43:05.284123 17577 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:05.284403 17577 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:05.284945 17577 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:43:05.286202 17087 ts_manager.cc:194] Registered new tserver with Master: 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
I20260806 02:43:05.286851 17087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:60425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:43:05.303972 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:05.415480 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:05.415716 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:05.415756 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:05.419714 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:05.419771 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:05.419864 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:43:05.425014 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:05.426017 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:05.426975 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:05.437022 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.437016 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.437161 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:05.437252 17585 server_base.cc:1034] running on GCE node
I20260806 02:43:05.437817 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:05.438417 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:05.439608 17585 hybrid_clock.cc:648] HybridClock initialized: now 1785984185439575 us; error 49 us; skew 500 ppm
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:05.442435 17585 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:43:05.443419 17585 webserver.cc:460] Webserver started at http://127.28.191.3:35765/ using document root <none> and password file <none>
I20260806 02:43:05.444002 17585 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:05.444092 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:05.444317 17585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:05.446390 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "6ca7fdb736474ca98898b01b36fa7853"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "79dc8b49cc40a29642414c82f282bafa"
server_key_iv: "993c0116a1dc04e29d4e4db4fde57b1b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.446971 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "6ca7fdb736474ca98898b01b36fa7853"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "79dc8b49cc40a29642414c82f282bafa"
server_key_iv: "993c0116a1dc04e29d4e4db4fde57b1b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.450773 17585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:43:05.453001 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.454212 17585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:05.454360 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "6ca7fdb736474ca98898b01b36fa7853"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "79dc8b49cc40a29642414c82f282bafa"
server_key_iv: "993c0116a1dc04e29d4e4db4fde57b1b"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.454468 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:05.469911 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:05.473024 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:05.473242 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:05.473800 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:05.474628 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:05.474697 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.474766 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:05.474818 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.483742 17585 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:37139
I20260806 02:43:05.483771 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:37139 every 8 connection(s)
I20260806 02:43:05.484613 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:43:05.490787 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17585
I20260806 02:43:05.490937 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:43:05.491214 29436 external_mini_cluster.cc:1428] Setting key 53f6a163e66a88bc686b66a8d8a890d0
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:05.498291 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.486444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:44321 (local address 127.28.191.62:45297)
0806 02:43:05.486588 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.486592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.487377 (+ 785us) server_negotiation.cc:407] Connection header received
0806 02:43:05.488737 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.488740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.488802 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.488882 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:05.490104 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.490646 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.491988 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.492266 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.492378 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.495192 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:05.495195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:05.495197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:05.495220 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:05.496470 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.496943 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.496945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.496947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.497019 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.497380 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.497382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.497383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.497558 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:05.497635 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.498082 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.498177 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:43:05.499045 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.486785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:44321)
0806 02:43:05.487217 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.487258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.488511 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.489046 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.489053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.489436 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:05.489935 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.489949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.490777 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.490788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.491855 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.491865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.492413 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.492416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.492646 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.493301 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:05.493317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:05.495045 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:05.496580 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.496585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.496596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.496857 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.497166 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:05.497168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:05.497169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:05.497297 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:05.497648 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:05.497653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:05.497869 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.498020 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.498256 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260806 02:43:05.500203 17776 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:05.500465 17776 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:05.500948 17776 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
I20260806 02:43:05.501958 17087 ts_manager.cc:194] Registered new tserver with Master: 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139)
I20260806 02:43:05.502475 17087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:44321
I20260806 02:43:05.505198 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:43:05.513937 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.507147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.28.191.62:45297)
0806 02:43:05.507260 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.507263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.507412 (+ 149us) server_negotiation.cc:407] Connection header received
0806 02:43:05.507494 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.507495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.507529 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.507595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:05.508262 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.508726 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.509562 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.509761 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.509836 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.511098 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:05.511100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:05.511102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:05.511118 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:05.512328 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.512676 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.512678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.512680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.512745 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:05.513154 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:05.513157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:05.513158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:05.513352 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:05.513453 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.513714 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.513808 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:43:05.517004 17087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:43:05.520149 17087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:43:05.534762 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.529185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32909 (local address 127.0.0.1:35262)
0806 02:43:05.530173 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.530186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.530308 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.530769 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.530771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.530787 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:05.530873 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.530881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.531825 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.531828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.533240 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.533252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534208 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.534336 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.534377 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.534495 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.534616 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":34,"thread_start_us":103,"threads_started":1}
I20260806 02:43:05.534775 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.529366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.28.191.2:32909)
0806 02:43:05.529948 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.529960 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.530287 (+ 327us) server_negotiation.cc:407] Connection header received
0806 02:43:05.530440 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.530445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.530622 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.530767 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:05.530967 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.531684 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.533368 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534080 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534206 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.534416 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.534530 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.534619 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":116,"threads_started":1}
I20260806 02:43:05.535346 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.528590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37139 (local address 127.0.0.1:59040)
0806 02:43:05.529769 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.529783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.529938 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.530575 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.530577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.530592 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:05.530673 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.530677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.532200 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.532203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.533354 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.533365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534626 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.534822 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.534848 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.535141 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.535214 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":44,"thread_start_us":112,"threads_started":1}
I20260806 02:43:05.535353 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.528797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.28.191.3:37139)
0806 02:43:05.529225 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.529239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.529866 (+ 627us) server_negotiation.cc:407] Connection header received
0806 02:43:05.530129 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.530134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.530299 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.530425 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:05.530829 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.532001 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.533523 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534313 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534425 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.534557 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.535162 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.535242 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":173,"threads_started":1}
I20260806 02:43:05.536031 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.528471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40593 (local address 127.0.0.1:44402)
0806 02:43:05.529563 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.529614 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.529745 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.530446 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.530451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.530473 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:05.530626 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.530634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.532172 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.532176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.533325 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.533337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534626 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.534826 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.535762 (+ 936us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.535836 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.535905 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":56,"thread_start_us":95,"threads_started":1}
I20260806 02:43:05.536502 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.529294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.28.191.1:40593)
0806 02:43:05.529710 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.529725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.529750 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:43:05.529946 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.529953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.530180 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.530332 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:05.530758 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.532001 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.533445 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.534313 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.534425 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.534567 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.536261 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.536364 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260806 02:43:05.538022 17673 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea3552abde64268a9e295dfe35f05dd (DEFAULT_TABLE table=test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:05.537963 17460 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea3552abde64268a9e295dfe35f05dd (DEFAULT_TABLE table=test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:05.538986 17673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea3552abde64268a9e295dfe35f05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:05.539017 17460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea3552abde64268a9e295dfe35f05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:05.539041 17269 tablet_service.cc:1467] Processing CreateTablet for tablet 0ea3552abde64268a9e295dfe35f05dd (DEFAULT_TABLE table=test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:05.539964 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ea3552abde64268a9e295dfe35f05dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:05.546545 17800 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Bootstrap starting.
I20260806 02:43:05.548877 17800 tablet_bootstrap.cc:654] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:05.549468 17799 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Bootstrap starting.
I20260806 02:43:05.549748 17800 log.cc:824] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:05.550776 17801 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap starting.
I20260806 02:43:05.551936 17799 tablet_bootstrap.cc:654] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:05.551985 17800 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: No bootstrap required, opened a new log
I20260806 02:43:05.552163 17800 ts_tablet_manager.cc:1397] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 02:43:05.552695 17799 log.cc:824] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:05.552791 17801 tablet_bootstrap.cc:654] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:05.553584 17801 log.cc:824] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:05.554575 17799 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: No bootstrap required, opened a new log
I20260806 02:43:05.554777 17799 ts_tablet_manager.cc:1397] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260806 02:43:05.555434 17801 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: No bootstrap required, opened a new log
I20260806 02:43:05.555616 17801 ts_tablet_manager.cc:1397] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:43:05.556084 17800 raft_consensus.cc:348] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.556349 17800 raft_consensus.cc:374] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:05.556403 17800 raft_consensus.cc:729] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7f7184e5f145ce9c39f7febc13fdad, State: Initialized, Role: FOLLOWER
I20260806 02:43:05.557096 17800 consensus_queue.cc:260] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.558154 17800 ts_tablet_manager.cc:1428] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:43:05.558365 17799 raft_consensus.cc:348] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.558547 17577 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
I20260806 02:43:05.558609 17799 raft_consensus.cc:374] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:05.558668 17799 raft_consensus.cc:729] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca7fdb736474ca98898b01b36fa7853, State: Initialized, Role: FOLLOWER
I20260806 02:43:05.559149 17799 consensus_queue.cc:260] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.560037 17799 ts_tablet_manager.cc:1428] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260806 02:43:05.559900 17801 raft_consensus.cc:348] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.560143 17801 raft_consensus.cc:374] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:05.560213 17801 raft_consensus.cc:729] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6443500a749e18ad8b4fe2de7cf82, State: Initialized, Role: FOLLOWER
I20260806 02:43:05.560379 17776 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
I20260806 02:43:05.560648 17801 consensus_queue.cc:260] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.561365 17361 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
I20260806 02:43:05.562911 17801 ts_tablet_manager.cc:1428] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260806 02:43:05.624452 17807 raft_consensus.cc:482] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:05.624751 17807 raft_consensus.cc:504] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.625988 17807 leader_election.cc:288] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593), 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139)
I20260806 02:43:05.630765 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.626270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37139 (local address 127.28.191.2:52463)
0806 02:43:05.626419 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.626437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.626536 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.626841 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.626843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.626867 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:05.627009 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.627017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.628060 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.628063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.629335 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.629346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.630281 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.630286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.630392 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.630418 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.630520 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.630619 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":2}
I20260806 02:43:05.630771 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.626394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:52463 (local address 127.28.191.3:37139)
0806 02:43:05.626521 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.626528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.626548 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:43:05.626661 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.626664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.626717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.626811 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:05.627137 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.627919 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.629486 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.630138 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.630274 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.630355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.630539 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.630619 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:43:05.630929 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.626367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:38633 (local address 127.28.191.1:40593)
0806 02:43:05.626505 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.626510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.626942 (+ 432us) server_negotiation.cc:407] Connection header received
0806 02:43:05.627104 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.627108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.627163 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.627241 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:05.627491 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.628159 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.629620 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.630267 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.630378 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.630429 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.630734 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.630817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:43:05.631493 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.626271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40593 (local address 127.28.191.2:38633)
0806 02:43:05.626852 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.626864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:05.626953 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:05.627264 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:05.627266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:05.627282 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:05.627377 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.627382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.628273 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.628277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.629466 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:05.629476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.630433 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.630438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:05.630587 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.630610 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:43:05.631104 (+ 494us) client_negotiation.cc:240] Negotiation successful
0806 02:43:05.631375 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":181,"thread_start_us":166,"threads_started":1}
I20260806 02:43:05.631487 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea3552abde64268a9e295dfe35f05dd" candidate_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca7fdb736474ca98898b01b36fa7853" is_pre_election: true
I20260806 02:43:05.631896 17702 raft_consensus.cc:2454] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7f7184e5f145ce9c39f7febc13fdad in term 0.
I20260806 02:43:05.632229 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea3552abde64268a9e295dfe35f05dd" candidate_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" is_pre_election: true
I20260806 02:43:05.632421 17395 leader_election.cc:302] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853; no voters:
I20260806 02:43:05.632546 17296 raft_consensus.cc:2454] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7f7184e5f145ce9c39f7febc13fdad in term 0.
I20260806 02:43:05.632754 17807 raft_consensus.cc:2781] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:43:05.632845 17807 raft_consensus.cc:482] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:43:05.632903 17807 raft_consensus.cc:3037] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:05.634043 17807 raft_consensus.cc:504] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.634433 17807 leader_election.cc:288] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [CANDIDATE]: Term 1 election: Requested vote from peers 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593), 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139)
I20260806 02:43:05.634811 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea3552abde64268a9e295dfe35f05dd" candidate_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
I20260806 02:43:05.634820 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ea3552abde64268a9e295dfe35f05dd" candidate_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca7fdb736474ca98898b01b36fa7853"
I20260806 02:43:05.634907 17296 raft_consensus.cc:3037] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:05.634949 17702 raft_consensus.cc:3037] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:05.635923 17296 raft_consensus.cc:2454] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7f7184e5f145ce9c39f7febc13fdad in term 1.
I20260806 02:43:05.636018 17702 raft_consensus.cc:2454] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7f7184e5f145ce9c39f7febc13fdad in term 1.
I20260806 02:43:05.636245 17398 leader_election.cc:302] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7f7184e5f145ce9c39f7febc13fdad, 7fb6443500a749e18ad8b4fe2de7cf82; no voters:
I20260806 02:43:05.636416 17807 raft_consensus.cc:2781] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:05.636622 17807 raft_consensus.cc:686] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 LEADER]: Becoming Leader. State: Replica: 1e7f7184e5f145ce9c39f7febc13fdad, State: Running, Role: LEADER
I20260806 02:43:05.636947 17807 consensus_queue.cc:237] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:05.639992 17091 catalog_manager.cc:5495] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2). New cstate: current_term: 1 leader_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260806 02:43:05.662789 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.659156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.28.191.2:32909)
0806 02:43:05.659300 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:05.659305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:05.659320 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:43:05.659426 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:05.659428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:05.659482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:05.659609 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:05.659857 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.660359 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.661295 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:05.661513 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:05.661634 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:05.661950 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:05.662352 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:43:05.662490 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:05.662566 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 02:43:05.662620 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:43:05.668977 17296 raft_consensus.cc:1261] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Refusing update from remote peer 1e7f7184e5f145ce9c39f7febc13fdad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:05.669023 17702 raft_consensus.cc:1261] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Refusing update from remote peer 1e7f7184e5f145ce9c39f7febc13fdad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:05.669867 17813 consensus_queue.cc:1035] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:05.670117 17807 consensus_queue.cc:1035] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:05.677868 17817 mvcc.cc:205] Tried to move back new op lower bound from 7315391224491515904 to 7315391224371458048. Current Snapshot: MvccSnapshot[applied={T|T < 7315391224491515904}]
I20260806 02:43:05.680418 17818 mvcc.cc:205] Tried to move back new op lower bound from 7315391224491515904 to 7315391224371458048. Current Snapshot: MvccSnapshot[applied={T|T < 7315391224491515904}]
I20260806 02:43:05.687295 17091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:05.687481 17091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:05.689811 17091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Sending DeleteTablet for 3 replicas of tablet 0ea3552abde64268a9e295dfe35f05dd
I20260806 02:43:05.690673 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17187
I20260806 02:43:05.690660 17673 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea3552abde64268a9e295dfe35f05dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:05 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:59040
I20260806 02:43:05.694980 17460 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea3552abde64268a9e295dfe35f05dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:05 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:35262
I20260806 02:43:05.696429 17828 tablet_replica.cc:330] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: stopping tablet replica
I20260806 02:43:05.696787 17828 raft_consensus.cc:2229] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:05.697165 17828 raft_consensus.cc:2258] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:05.697569 17830 tablet_replica.cc:330] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: stopping tablet replica
I20260806 02:43:05.697829 17830 raft_consensus.cc:2229] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:43:05.698187 17830 raft_consensus.cc:2258] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:05.700348 17830 ts_tablet_manager.cc:1905] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:05.701390 17828 ts_tablet_manager.cc:1905] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:05.703135 17830 ts_tablet_manager.cc:1918] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:05.703231 17830 log.cc:1196] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/0ea3552abde64268a9e295dfe35f05dd
I20260806 02:43:05.703558 17830 ts_tablet_manager.cc:1939] T 0ea3552abde64268a9e295dfe35f05dd P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting consensus metadata
I20260806 02:43:05.704741 17828 ts_tablet_manager.cc:1918] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:05.704802 17075 catalog_manager.cc:4841] TS 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): tablet 0ea3552abde64268a9e295dfe35f05dd (table test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]) successfully deleted
I20260806 02:43:05.704860 17828 log.cc:1196] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/0ea3552abde64268a9e295dfe35f05dd
I20260806 02:43:05.705404 17828 ts_tablet_manager.cc:1939] T 0ea3552abde64268a9e295dfe35f05dd P 6ca7fdb736474ca98898b01b36fa7853: Deleting consensus metadata
I20260806 02:43:05.706642 17072 catalog_manager.cc:4841] TS 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): tablet 0ea3552abde64268a9e295dfe35f05dd (table test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]) successfully deleted
W20260806 02:43:05.708009 17075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:40593 (error 108)
I20260806 02:43:05.708870 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17371
I20260806 02:43:05.709126 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:05.708790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40593 (local address 127.0.0.1:44404)
0806 02:43:05.708926 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:05.709000 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:43:05.711539 17075 catalog_manager.cc:4584] TS 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ea3552abde64268a9e295dfe35f05dd: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40593: connect: Connection refused (error 111)
I20260806 02:43:05.717141 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17585
I20260806 02:43:05.727058 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17051
I20260806 02:43:05.737140 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--webserver_interface=127.28.191.62
--webserver_port=38549
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:05.849851 17832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:05.850221 17832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:05.850268 17832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:05.854231 17832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:05.854282 17832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:05.854302 17832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:05.854319 17832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:05.854334 17832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:05.859488 17832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=38549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:05.860467 17832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:05.861443 17832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:05.874833 17839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.874895 17841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:05.874852 17838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:05.875170 17832 server_base.cc:1034] running on GCE node
I20260806 02:43:05.875803 17832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:05.877027 17832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:05.878247 17832 hybrid_clock.cc:648] HybridClock initialized: now 1785984185878228 us; error 29 us; skew 500 ppm
Aug 06 02:43:05 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:05.881210 17832 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:05.882143 17832 webserver.cc:460] Webserver started at http://127.28.191.62:38549/ using document root <none> and password file <none>
I20260806 02:43:05.882791 17832 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:05.882928 17832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:05.886955 17832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:05.889029 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:05.890295 17832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:43:05.890440 17832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "20969d1df7d64233b1981b8bdfcc6a70"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9648528a23482b433dba3c31bdaef089"
server_key_iv: "6ce9e71bcde1faff270fe2027fb4b774"
server_key_version: "encryptionkey@0"
I20260806 02:43:05.891055 17832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:05.918834 17832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:05.921749 17832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:05.921964 17832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:05.928812 17832 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:45297
I20260806 02:43:05.929176 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:45297 every 8 connection(s)
I20260806 02:43:05.929949 17832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:05.933805 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17832
I20260806 02:43:05.934312 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:40593
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=35623
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:05.934497 17937 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:43:05.936374 17937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap starting.
I20260806 02:43:05.954121 17937 log.cc:824] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:05.962867 17937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:05.963227 17937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap complete.
I20260806 02:43:05.967473 17937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:05.967860 17937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Initialized, Role: FOLLOWER
I20260806 02:43:05.968344 17937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:05.968459 17937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:05.968564 17937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:05.968676 17937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:43:05.970714 17937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:05.971058 17937 leader_election.cc:302] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20969d1df7d64233b1981b8bdfcc6a70; no voters:
I20260806 02:43:05.971402 17937 leader_election.cc:288] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:43:05.971493 17941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:43:05.971693 17941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 LEADER]: Becoming Leader. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Running, Role: LEADER
I20260806 02:43:05.972005 17941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:05.972658 17937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:05.972916 17941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20969d1df7d64233b1981b8bdfcc6a70. Latest consensus state: current_term: 2 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:05.973053 17941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:05.973469 17942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:05.973644 17942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:05.974704 17949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:05.982319 17949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:05.983115 17949 catalog_manager.cc:1259] Loaded cluster ID: ff1ac29959d347cf955d5284c9348b61
I20260806 02:43:05.983220 17949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:43:05.984697 17949 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:43:05.989231 17949 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:43:05.996270 17949 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:05.997231 17949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Loaded TSK: 0
I20260806 02:43:05.998520 17949 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:43:06.059693 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:06.059948 17939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:06.059990 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:06.064093 17939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:06.064148 17939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:06.064226 17939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:43:06.069469 17939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=35623
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:06.070533 17939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:06.071727 17939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:06.081511 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.081414 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.081399 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:06.081918 17939 server_base.cc:1034] running on GCE node
I20260806 02:43:06.082329 17939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:06.082939 17939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:06.084128 17939 hybrid_clock.cc:648] HybridClock initialized: now 1785984186084098 us; error 50 us; skew 500 ppm
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:06.087147 17939 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:43:06.088112 17939 webserver.cc:460] Webserver started at http://127.28.191.1:35623/ using document root <none> and password file <none>
I20260806 02:43:06.088680 17939 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:06.088753 17939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:06.092701 17939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:43:06.095122 17989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.096441 17939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:43:06.096585 17939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9677ec1ebae9fa51c71262cac9071557"
server_key_iv: "08480c7e6b07595d6d4d40f3fdc9c4ef"
server_key_version: "encryptionkey@0"
I20260806 02:43:06.097218 17939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:06.129650 17939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:06.133118 17939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:06.133428 17939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:06.134079 17939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:06.135244 17998 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:43:06.139531 17939 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:43:06.139616 17939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 02:43:06.139703 17939 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:43:06.141371 17939 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:43:06.141445 17939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:06.141700 17998 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap starting.
I20260806 02:43:06.152343 17939 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:40593
I20260806 02:43:06.152406 18154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:40593 every 8 connection(s)
I20260806 02:43:06.153275 17939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:43:06.161643 17998 log.cc:824] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:06.163050 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 17939
I20260806 02:43:06.163522 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:32909
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=40597
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:06.170989 17998 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:06.171305 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.155502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:47205 (local address 127.28.191.62:45297)
0806 02:43:06.156020 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.156040 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.156120 (+ 80us) server_negotiation.cc:407] Connection header received
0806 02:43:06.157285 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.157301 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.157753 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.158185 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:06.159346 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.160851 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.162150 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.162706 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.162904 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.166272 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:06.166277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:06.166289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:06.166316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:06.168070 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.168644 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.168650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.168662 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.168776 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.169173 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.169178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.169180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.169524 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:06.169669 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.170261 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.170525 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":141,"threads_started":1}
I20260806 02:43:06.171319 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.155449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:47205)
0806 02:43:06.155956 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.156017 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.156824 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.158207 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.158213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.158555 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:06.159143 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.159170 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.161020 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.161023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.161975 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.162032 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.163098 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.163102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.163364 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.164168 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:06.164193 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:06.166075 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:06.168234 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.168239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.168251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.168525 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.168877 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.168881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.168883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.169026 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.169643 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:06.169649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:06.169885 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.170102 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.170429 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":151,"threads_started":1}
I20260806 02:43:06.171466 17998 tablet_bootstrap.cc:492] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap complete.
I20260806 02:43:06.173015 17998 ts_tablet_manager.cc:1397] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.004s
I20260806 02:43:06.173820 18156 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:06.174043 18156 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:06.174489 18156 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
I20260806 02:43:06.176739 17876 ts_manager.cc:194] Registered new tserver with Master: 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593)
I20260806 02:43:06.176985 17998 raft_consensus.cc:348] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:06.177670 17998 raft_consensus.cc:729] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6443500a749e18ad8b4fe2de7cf82, State: Initialized, Role: FOLLOWER
I20260806 02:43:06.178193 17998 consensus_queue.cc:260] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } } peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } }
I20260806 02:43:06.179160 17998 ts_tablet_manager.cc:1428] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:43:06.180512 17876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:47205
I20260806 02:43:06.182976 18156 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:43:06.190824 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.179867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40593 (local address 127.0.0.1:44406)
0806 02:43:06.180584 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.180633 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.180781 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.181301 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.181307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.181710 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:06.181834 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.181842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.183068 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.183071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.184009 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.184020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.184761 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.184767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.184890 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.185412 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:06.185428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:06.187047 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:06.188894 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.188904 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.188909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.189291 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.189617 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.189621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.189623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.189714 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.190153 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:06.190159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:06.190296 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.190437 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.190651 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":100,"threads_started":1}
I20260806 02:43:06.190855 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.179958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.28.191.1:40593)
0806 02:43:06.180321 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.180332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.180727 (+ 395us) server_negotiation.cc:407] Connection header received
0806 02:43:06.180922 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.180927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.181137 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.181267 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:06.181964 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.182940 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.184175 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.184639 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.184722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.187186 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:06.187194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:06.187203 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:06.187223 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:06.188785 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.189407 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.189411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.189412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.189514 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.189815 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.189825 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.189826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.190038 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:06.190174 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.190537 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.190709 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260806 02:43:06.191733 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ea3552abde64268a9e295dfe35f05dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:05 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:44406
I20260806 02:43:06.192512 18169 tablet_replica.cc:330] stopping tablet replica
I20260806 02:43:06.192790 18169 raft_consensus.cc:2229] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:06.193023 18169 raft_consensus.cc:2258] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:06.194077 18169 ts_tablet_manager.cc:1905] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:06.197642 18169 ts_tablet_manager.cc:1918] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:06.197722 18169 log.cc:1196] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/0ea3552abde64268a9e295dfe35f05dd
I20260806 02:43:06.198024 18169 ts_tablet_manager.cc:1939] T 0ea3552abde64268a9e295dfe35f05dd P 7fb6443500a749e18ad8b4fe2de7cf82: Deleting consensus metadata
I20260806 02:43:06.199645 17857 catalog_manager.cc:4841] TS 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593): tablet 0ea3552abde64268a9e295dfe35f05dd (table test-table [id=3464424cba84462dbc0eaf8bb0d5f65d]) successfully deleted
W20260806 02:43:06.286659 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:06.286904 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:06.286947 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:06.290868 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:06.290921 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:06.290978 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:43:06.296355 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:32909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=40597
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:06.297470 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:06.298470 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:43:06.308523 18164 server_base.cc:1034] running on GCE node
W20260806 02:43:06.308475 18176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.308468 18177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.308492 18180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:06.309198 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:06.309784 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:06.310940 18164 hybrid_clock.cc:648] HybridClock initialized: now 1785984186310910 us; error 45 us; skew 500 ppm
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:06.313874 18164 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:43:06.314769 18164 webserver.cc:460] Webserver started at http://127.28.191.2:40597/ using document root <none> and password file <none>
I20260806 02:43:06.315335 18164 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:06.315440 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:06.319219 18164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 02:43:06.321295 18188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.322460 18164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:06.322615 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "de60135ccf99aff78a444d73fea73c45"
server_key_iv: "a975fa87570c5b765d1b0db28465052b"
server_key_version: "encryptionkey@0"
I20260806 02:43:06.323140 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:06.357255 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:06.360456 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:06.360685 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:06.361351 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:06.362279 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:06.362351 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.362421 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:06.362475 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.371477 18164 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:32909
I20260806 02:43:06.371503 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:32909 every 8 connection(s)
I20260806 02:43:06.372648 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:06.382616 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 18164
I20260806 02:43:06.383121 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:37139
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=35765
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:06.386816 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.374564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:36691 (local address 127.28.191.62:45297)
0806 02:43:06.374731 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.374736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.375528 (+ 792us) server_negotiation.cc:407] Connection header received
0806 02:43:06.376937 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.376941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.377000 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.377144 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:06.378333 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.378867 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.380016 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.380257 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.380371 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.382941 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:06.382945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:06.382946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:06.382973 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:06.384405 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.385088 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.385094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.385097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.385207 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.385555 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.385560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.385563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.385823 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:06.385935 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.386435 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.386608 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:43:06.387509 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.374955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:36691)
0806 02:43:06.375380 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.375423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.376711 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.377330 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.377337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.377683 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:06.378164 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.378177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.379002 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.379005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.379888 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.379898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.380422 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.380426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.380612 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.381184 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:06.381201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:06.382767 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:06.384526 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.384534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.384550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.384932 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.385307 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.385310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.385311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.385448 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.385934 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:06.385939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:06.386181 (+ 242us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.386339 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.386592 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260806 02:43:06.388811 18354 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:06.389055 18354 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:06.389578 18354 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
I20260806 02:43:06.390688 17876 ts_manager.cc:194] Registered new tserver with Master: 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
I20260806 02:43:06.391247 17876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:36691
W20260806 02:43:06.497535 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:06.497781 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:06.497822 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:06.502274 18359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:06.502329 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:06.502430 18359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:43:06.508244 18359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:37139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=35765
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:06.509301 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:06.510389 18359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:06.520253 18366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.520299 18368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.520299 18365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:06.520696 18359 server_base.cc:1034] running on GCE node
I20260806 02:43:06.521134 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:06.521732 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:06.522936 18359 hybrid_clock.cc:648] HybridClock initialized: now 1785984186522911 us; error 44 us; skew 500 ppm
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:06.525708 18359 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:43:06.526630 18359 webserver.cc:460] Webserver started at http://127.28.191.3:35765/ using document root <none> and password file <none>
I20260806 02:43:06.527169 18359 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:06.527271 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:06.531174 18359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:43:06.533149 18379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.534399 18359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:06.534551 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "6ca7fdb736474ca98898b01b36fa7853"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "79dc8b49cc40a29642414c82f282bafa"
server_key_iv: "993c0116a1dc04e29d4e4db4fde57b1b"
server_key_version: "encryptionkey@0"
I20260806 02:43:06.535125 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:06.557228 18359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:06.560115 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:06.560323 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:06.560930 18359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:06.561806 18359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:06.561853 18359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.561892 18359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:06.561908 18359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.570870 18359 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:37139
I20260806 02:43:06.570943 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:37139 every 8 connection(s)
I20260806 02:43:06.571935 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:43:06.581516 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 18359
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:06.586129 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.573821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:58153 (local address 127.28.191.62:45297)
0806 02:43:06.574013 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.574018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.574910 (+ 892us) server_negotiation.cc:407] Connection header received
0806 02:43:06.576289 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.576293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.576354 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.576461 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:06.577648 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.578174 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.579307 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.579539 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.579656 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.582317 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:06.582324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:06.582330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:06.582363 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:06.583935 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.584516 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.584521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.584523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.584619 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.584950 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.584953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.584956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.585213 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:06.585313 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.585833 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.585935 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:43:06.587005 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.574319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:58153)
0806 02:43:06.574757 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.574798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.576071 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.576643 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.576650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.577008 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:06.577489 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.577502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.578303 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.578307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.579174 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.579184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.579694 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.579699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.579870 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.580469 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:06.580485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:06.582115 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:06.584048 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.584054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.584068 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.584391 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.584717 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:06.584719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:06.584720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:06.584852 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:06.585312 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:06.585317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:06.585541 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.585765 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.586136 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260806 02:43:06.588447 18533 heartbeater.cc:347] Connected to a master server at 127.28.191.62:45297
I20260806 02:43:06.588750 18533 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:06.589325 18533 heartbeater.cc:511] Master 127.28.191.62:45297 requested a full tablet report, sending...
I20260806 02:43:06.590513 17876 ts_manager.cc:194] Registered new tserver with Master: 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139)
I20260806 02:43:06.591203 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.582258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.28.191.62:45297)
0806 02:43:06.582541 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.582545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.582647 (+ 102us) server_negotiation.cc:407] Connection header received
0806 02:43:06.582808 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.582811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.582853 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.582952 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:06.583724 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.584423 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.585618 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.585943 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.586047 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.587532 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:06.587536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:06.587538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:06.587564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:06.589305 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.589795 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.589797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.589798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.589873 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:06.590324 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:06.590327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:06.590329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:06.590559 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:06.590758 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.590936 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.591052 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260806 02:43:06.591223 17876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:58153
I20260806 02:43:06.592993 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:43:06.597185 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.593639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.28.191.62:45297)
0806 02:43:06.593785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.593789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.593962 (+ 173us) server_negotiation.cc:407] Connection header received
0806 02:43:06.594136 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.594139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.594218 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.594323 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:06.594581 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.595171 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.595961 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.596180 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.596259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.596523 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:06.596854 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:43:06.596939 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.596999 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.597045 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:43:06.597760 17876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:43:06.600942 17876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:43:06.608878 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 936332885ca540138dfcc7d6a26c0b09 (DEFAULT_TABLE table=test-table [id=fe648ff23ab749d7b7cac0d5bacc5969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:06.609553 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936332885ca540138dfcc7d6a26c0b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:06.612556 17998 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap starting.
I20260806 02:43:06.613214 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.607428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37139 (local address 127.0.0.1:36462)
0806 02:43:06.607564 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.607576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.607638 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.608550 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.608553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.608570 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:06.608674 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.608684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.610300 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.610304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.611517 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.611529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.612757 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.612761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.612855 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.612905 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.612962 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.613038 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 02:43:06.613355 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.607890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32909 (local address 127.0.0.1:38822)
0806 02:43:06.608473 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.608487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.608587 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.609421 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.609424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.609446 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:06.609537 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.609543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.610799 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.610804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.611559 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.611568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.612949 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.612954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.613049 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.613093 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.613161 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.613222 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":101,"threads_started":1}
I20260806 02:43:06.613523 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.608092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.28.191.2:32909)
0806 02:43:06.608942 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.608957 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.608979 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:43:06.609053 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.609082 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.609270 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.609421 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:06.609638 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.610661 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.611721 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.612614 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.612733 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.612884 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.613225 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.613328 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":97,"threads_started":1}
I20260806 02:43:06.613523 17998 tablet_bootstrap.cc:654] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:06.614142 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.607587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.28.191.3:37139)
0806 02:43:06.607911 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.607923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.607945 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:43:06.608022 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.608028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.608235 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.608836 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:06.608976 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.610155 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.611666 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.612617 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.613805 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.613913 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.613955 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.614030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260806 02:43:06.616499 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 936332885ca540138dfcc7d6a26c0b09 (DEFAULT_TABLE table=test-table [id=fe648ff23ab749d7b7cac0d5bacc5969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:06.616503 18264 tablet_service.cc:1467] Processing CreateTablet for tablet 936332885ca540138dfcc7d6a26c0b09 (DEFAULT_TABLE table=test-table [id=fe648ff23ab749d7b7cac0d5bacc5969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:06.617570 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936332885ca540138dfcc7d6a26c0b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:06.617584 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936332885ca540138dfcc7d6a26c0b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:06.617971 17998 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: No bootstrap required, opened a new log
I20260806 02:43:06.618065 17998 ts_tablet_manager.cc:1397] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:43:06.618513 17998 raft_consensus.cc:348] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.618635 17998 raft_consensus.cc:374] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:06.618673 17998 raft_consensus.cc:729] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6443500a749e18ad8b4fe2de7cf82, State: Initialized, Role: FOLLOWER
I20260806 02:43:06.618806 17998 consensus_queue.cc:260] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.619065 17998 ts_tablet_manager.cc:1428] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:43:06.625415 18558 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Bootstrap starting.
I20260806 02:43:06.625690 18555 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Bootstrap starting.
I20260806 02:43:06.628033 18558 tablet_bootstrap.cc:654] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:06.628060 18555 tablet_bootstrap.cc:654] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:06.628739 18555 log.cc:824] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:06.628741 18558 log.cc:824] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:06.630666 18558 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: No bootstrap required, opened a new log
I20260806 02:43:06.630673 18555 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: No bootstrap required, opened a new log
I20260806 02:43:06.630864 18558 ts_tablet_manager.cc:1397] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:43:06.630878 18555 ts_tablet_manager.cc:1397] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:43:06.633911 18555 raft_consensus.cc:348] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.634125 18555 raft_consensus.cc:374] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:06.634195 18555 raft_consensus.cc:729] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca7fdb736474ca98898b01b36fa7853, State: Initialized, Role: FOLLOWER
I20260806 02:43:06.634024 18558 raft_consensus.cc:348] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.634296 18558 raft_consensus.cc:374] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:06.634387 18558 raft_consensus.cc:729] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7f7184e5f145ce9c39f7febc13fdad, State: Initialized, Role: FOLLOWER
I20260806 02:43:06.634693 18555 consensus_queue.cc:260] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.634845 18558 consensus_queue.cc:260] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.635403 18533 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
I20260806 02:43:06.635703 18354 heartbeater.cc:503] Master 127.28.191.62:45297 was elected leader, sending a full tablet report...
I20260806 02:43:06.635730 18555 ts_tablet_manager.cc:1428] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260806 02:43:06.635807 18558 ts_tablet_manager.cc:1428] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 02:43:06.654461 18158 tablet.cc:2369] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:43:06.703812 18166 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:06.704037 18166 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.705255 18166 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
I20260806 02:43:06.709573 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.705436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32909 (local address 127.28.191.1:45393)
0806 02:43:06.705772 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.705796 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.705948 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.706275 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.706278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.706295 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:06.706476 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.706481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.707209 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.707213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.708263 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.708273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.709167 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.709171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.709248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.709288 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.709354 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.709410 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":44,"spinlock_wait_cycles":26496,"thread_start_us":96,"threads_started":1}
I20260806 02:43:06.709609 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.705621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:45393 (local address 127.28.191.2:32909)
0806 02:43:06.705767 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.705772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.705880 (+ 108us) server_negotiation.cc:407] Connection header received
0806 02:43:06.706074 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.706077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.706132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.706222 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:06.706604 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.707100 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.708399 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.709007 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.709168 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.709210 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.709396 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.709479 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:43:06.709792 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.705436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37139 (local address 127.28.191.1:42957)
0806 02:43:06.705806 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.705820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:06.705962 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:06.706326 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:06.706330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:06.706374 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:06.706485 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.706490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.707223 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.707227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.708286 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:06.708295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.709311 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.709314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:06.709421 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.709443 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:43:06.709519 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:43:06.709595 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":91,"spinlock_wait_cycles":1792,"thread_start_us":73,"threads_started":1}
I20260806 02:43:06.709792 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.705631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:42957 (local address 127.28.191.3:37139)
0806 02:43:06.705764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.705769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.705917 (+ 148us) server_negotiation.cc:407] Connection header received
0806 02:43:06.706102 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.706110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.706152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.706275 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:06.706606 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.707106 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.708446 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.709032 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.709154 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.709210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.709565 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.709656 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:43:06.710260 18291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936332885ca540138dfcc7d6a26c0b09" candidate_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" is_pre_election: true
I20260806 02:43:06.710352 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936332885ca540138dfcc7d6a26c0b09" candidate_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca7fdb736474ca98898b01b36fa7853" is_pre_election: true
I20260806 02:43:06.710628 18291 raft_consensus.cc:2454] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb6443500a749e18ad8b4fe2de7cf82 in term 0.
I20260806 02:43:06.710673 18474 raft_consensus.cc:2454] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb6443500a749e18ad8b4fe2de7cf82 in term 0.
I20260806 02:43:06.711159 17992 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7f7184e5f145ce9c39f7febc13fdad, 7fb6443500a749e18ad8b4fe2de7cf82; no voters:
I20260806 02:43:06.711445 18166 raft_consensus.cc:2781] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:43:06.711534 18166 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:43:06.711565 18166 raft_consensus.cc:3037] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:06.712620 18166 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.713019 18166 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 1 election: Requested vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
I20260806 02:43:06.713382 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936332885ca540138dfcc7d6a26c0b09" candidate_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca7fdb736474ca98898b01b36fa7853"
I20260806 02:43:06.713505 18291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936332885ca540138dfcc7d6a26c0b09" candidate_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
I20260806 02:43:06.713533 18474 raft_consensus.cc:3037] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:06.713644 18291 raft_consensus.cc:3037] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:06.714883 18291 raft_consensus.cc:2454] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb6443500a749e18ad8b4fe2de7cf82 in term 1.
I20260806 02:43:06.714883 18474 raft_consensus.cc:2454] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb6443500a749e18ad8b4fe2de7cf82 in term 1.
I20260806 02:43:06.715178 17992 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7f7184e5f145ce9c39f7febc13fdad, 7fb6443500a749e18ad8b4fe2de7cf82; no voters:
I20260806 02:43:06.715344 18166 raft_consensus.cc:2781] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:06.715543 18166 raft_consensus.cc:686] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 LEADER]: Becoming Leader. State: Replica: 7fb6443500a749e18ad8b4fe2de7cf82, State: Running, Role: LEADER
I20260806 02:43:06.715827 18166 consensus_queue.cc:237] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:06.718820 17876 catalog_manager.cc:5495] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1). New cstate: current_term: 1 leader_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } health_report { overall_health: HEALTHY } } }
I20260806 02:43:06.740789 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.737242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.28.191.1:40593)
0806 02:43:06.737509 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:06.737516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:06.737528 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:43:06.737598 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:06.737600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:06.737652 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:06.737752 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:06.738031 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.738529 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.739366 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:06.739580 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:06.739662 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:06.739950 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:06.740385 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:43:06.740477 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:06.740620 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 02:43:06.740671 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":116,"threads_started":1}
I20260806 02:43:06.744624 18291 raft_consensus.cc:1261] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Refusing update from remote peer 7fb6443500a749e18ad8b4fe2de7cf82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:06.744627 18474 raft_consensus.cc:1261] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Refusing update from remote peer 7fb6443500a749e18ad8b4fe2de7cf82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:06.745397 18166 consensus_queue.cc:1035] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:06.745699 18567 consensus_queue.cc:1035] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:06.752832 18572 mvcc.cc:205] Tried to move back new op lower bound from 7315391228899663872 to 7315391228790706176. Current Snapshot: MvccSnapshot[applied={T|T < 7315391228899663872}]
I20260806 02:43:06.754112 18573 mvcc.cc:205] Tried to move back new op lower bound from 7315391228899663872 to 7315391228790706176. Current Snapshot: MvccSnapshot[applied={T|T < 7315391228899663872}]
I20260806 02:43:06.761391 17875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:06.761538 17875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:06.763758 17875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Sending DeleteTablet for 3 replicas of tablet 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:06.764437 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 936332885ca540138dfcc7d6a26c0b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:06 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:36462
I20260806 02:43:06.764577 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17939
I20260806 02:43:06.764721 18264 tablet_service.cc:1514] Processing DeleteTablet for tablet 936332885ca540138dfcc7d6a26c0b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:06 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260806 02:43:06.773648 18587 tablet_replica.cc:330] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: stopping tablet replica
I20260806 02:43:06.773690 18586 tablet_replica.cc:330] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: stopping tablet replica
I20260806 02:43:06.774062 18586 raft_consensus.cc:2229] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:06.774063 18587 raft_consensus.cc:2229] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:06.774444 18587 raft_consensus.cc:2258] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:06.774444 18586 raft_consensus.cc:2258] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:06.776103 18587 ts_tablet_manager.cc:1905] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:06.776103 18586 ts_tablet_manager.cc:1905] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:06.778893 18587 ts_tablet_manager.cc:1918] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:06.778893 18586 ts_tablet_manager.cc:1918] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:06.779016 18586 log.cc:1196] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:06.779022 18587 log.cc:1196] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:06.779405 18586 ts_tablet_manager.cc:1939] T 936332885ca540138dfcc7d6a26c0b09 P 6ca7fdb736474ca98898b01b36fa7853: Deleting consensus metadata
I20260806 02:43:06.779405 18587 ts_tablet_manager.cc:1939] T 936332885ca540138dfcc7d6a26c0b09 P 1e7f7184e5f145ce9c39f7febc13fdad: Deleting consensus metadata
I20260806 02:43:06.780732 17851 catalog_manager.cc:4841] TS 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): tablet 936332885ca540138dfcc7d6a26c0b09 (table test-table [id=fe648ff23ab749d7b7cac0d5bacc5969]) successfully deleted
I20260806 02:43:06.780884 17857 catalog_manager.cc:4841] TS 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): tablet 936332885ca540138dfcc7d6a26c0b09 (table test-table [id=fe648ff23ab749d7b7cac0d5bacc5969]) successfully deleted
W20260806 02:43:06.781170 17857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:40593 (error 108)
I20260806 02:43:06.781214 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 18164
I20260806 02:43:06.788911 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:06.788609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:40593 (local address 127.0.0.1:47630)
0806 02:43:06.788759 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:06.788834 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 02:43:06.789112 17857 catalog_manager.cc:4584] TS 7fb6443500a749e18ad8b4fe2de7cf82 (127.28.191.1:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 936332885ca540138dfcc7d6a26c0b09: Network error: Client connection negotiation failed: client connection to 127.28.191.1:40593: connect: Connection refused (error 111)
I20260806 02:43:06.796744 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 18359
I20260806 02:43:06.805992 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 17832
I20260806 02:43:06.815411 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--webserver_interface=127.28.191.62
--webserver_port=38549
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:06.928062 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:06.928357 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:06.928447 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:06.932313 18589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:06.932394 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:06.932436 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:06.932494 18589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:06.932534 18589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:06.937621 18589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=38549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:06.938629 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:06.939603 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:06.947968 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.948007 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:06.948004 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:06.948066 18589 server_base.cc:1034] running on GCE node
I20260806 02:43:06.948724 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:06.949765 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:06.950922 18589 hybrid_clock.cc:648] HybridClock initialized: now 1785984186950897 us; error 43 us; skew 500 ppm
Aug 06 02:43:06 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:06.953688 18589 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:06.954506 18589 webserver.cc:460] Webserver started at http://127.28.191.62:38549/ using document root <none> and password file <none>
I20260806 02:43:06.954983 18589 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:06.955041 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:06.958711 18589 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 02:43:06.960747 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:06.961998 18589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 02:43:06.962131 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "20969d1df7d64233b1981b8bdfcc6a70"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9648528a23482b433dba3c31bdaef089"
server_key_iv: "6ce9e71bcde1faff270fe2027fb4b774"
server_key_version: "encryptionkey@0"
I20260806 02:43:06.962646 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:06.976358 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:06.979173 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:06.979388 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:06.986117 18589 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:45297
I20260806 02:43:06.986406 18670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:45297 every 8 connection(s)
I20260806 02:43:06.987212 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:06.991465 18672 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:43:06.991923 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 18589
I20260806 02:43:06.992386 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:40593
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=35623
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:06.993451 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap starting.
I20260806 02:43:07.012233 18672 log.cc:824] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:07.023841 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:07.024214 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70: Bootstrap complete.
I20260806 02:43:07.028398 18672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:07.028847 18672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Initialized, Role: FOLLOWER
I20260806 02:43:07.029382 18672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:07.029539 18672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:07.029630 18672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:07.029752 18672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:43:07.031585 18672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:07.031919 18672 leader_election.cc:302] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20969d1df7d64233b1981b8bdfcc6a70; no voters:
I20260806 02:43:07.032265 18672 leader_election.cc:288] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:43:07.032347 18680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:43:07.032536 18680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [term 3 LEADER]: Becoming Leader. State: Replica: 20969d1df7d64233b1981b8bdfcc6a70, State: Running, Role: LEADER
I20260806 02:43:07.032819 18680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } }
I20260806 02:43:07.033511 18672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:07.033814 18685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20969d1df7d64233b1981b8bdfcc6a70. Latest consensus state: current_term: 3 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:07.033941 18685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:07.034198 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "20969d1df7d64233b1981b8bdfcc6a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20969d1df7d64233b1981b8bdfcc6a70" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 45297 } } }
I20260806 02:43:07.034296 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20969d1df7d64233b1981b8bdfcc6a70 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:07.034358 18688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:07.039518 18688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:07.040380 18688 catalog_manager.cc:1259] Loaded cluster ID: ff1ac29959d347cf955d5284c9348b61
I20260806 02:43:07.040447 18688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:43:07.041586 18688 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9718bef9ad google::LogMessage::Fail() at ??:0
@ 0x7f9718bf288a google::LogMessage::SendToLog() at ??:0
@ 0x7f9718bef4de google::LogMessage::Flush() at ??:0
@ 0x7f9718bf0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f972352efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f972352fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f972352c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9723570640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f971ae413d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f971ae41c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f971ae433f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f971ae32512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f971c4766db start_thread at ??:0
@ 0x7f97176ed61f clone at ??:0
W20260806 02:43:07.121134 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:07.121362 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:07.121405 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:07.125689 18676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:07.125754 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:07.125850 18676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:43:07.131057 18676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=35623
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:07.132354 18676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:07.133502 18676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:07.143451 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:07.143438 18737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:07.143441 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:07.143951 18676 server_base.cc:1034] running on GCE node
I20260806 02:43:07.144356 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:07.145028 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:07.146242 18676 hybrid_clock.cc:648] HybridClock initialized: now 1785984187146210 us; error 45 us; skew 500 ppm
Aug 06 02:43:07 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:07.149147 18676 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:43:07.150102 18676 webserver.cc:460] Webserver started at http://127.28.191.1:35623/ using document root <none> and password file <none>
I20260806 02:43:07.150671 18676 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:07.150730 18676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:07.154703 18676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:07.156935 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.158380 18676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:43:07.158530 18676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "7fb6443500a749e18ad8b4fe2de7cf82"
format_stamp: "Formatted at 2026-08-06 02:43:04 on dist-test-slave-tfpf"
server_key: "9677ec1ebae9fa51c71262cac9071557"
server_key_iv: "08480c7e6b07595d6d4d40f3fdc9c4ef"
server_key_version: "encryptionkey@0"
I20260806 02:43:07.159102 18676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:07.174502 18676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:07.177765 18676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:07.177969 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:07.178529 18676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:07.179626 18753 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:43:07.185137 18676 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:43:07.185220 18676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 02:43:07.185302 18676 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:43:07.186976 18676 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:43:07.187049 18676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:43:07.187460 18753 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap starting.
I20260806 02:43:07.197877 18676 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:40593
I20260806 02:43:07.198076 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:40593 every 8 connection(s)
I20260806 02:43:07.198828 18676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:43:07.200814 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 18676
I20260806 02:43:07.201350 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:32909
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=40597
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:07.209502 18753 log.cc:824] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:07.218977 18753 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:07.219439 18753 tablet_bootstrap.cc:492] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Bootstrap complete.
I20260806 02:43:07.220893 18753 ts_tablet_manager.cc:1397] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20260806 02:43:07.224225 18753 raft_consensus.cc:348] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:07.224758 18753 raft_consensus.cc:729] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6443500a749e18ad8b4fe2de7cf82, State: Initialized, Role: FOLLOWER
I20260806 02:43:07.225296 18753 consensus_queue.cc:260] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:07.226367 18753 ts_tablet_manager.cc:1428] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260806 02:43:07.278445 18905 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:43:07.200873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:58485)
0806 02:43:07.202304 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.202361 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:07.203530 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:07.277496 (+ 73966us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":172,"threads_started":1}
W20260806 02:43:07.278975 18901 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:45297: Client connection negotiation failed: client connection to 127.28.191.62:45297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:43:07.279765 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.279479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:41267)
0806 02:43:07.279625 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.279699 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:43:07.280469 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.280235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:43405)
0806 02:43:07.280347 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.280403 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:43:07.280664 18901 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:43:07.326617 18907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:07.326853 18907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:07.326894 18907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:07.330811 18907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:07.330861 18907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:07.330964 18907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:43:07.336035 18907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:32909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=40597
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:07.337118 18907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:07.338089 18907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:07.347560 18922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:07.347570 18921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:07.347607 18907 server_base.cc:1034] running on GCE node
W20260806 02:43:07.347576 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:07.348160 18907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:07.348819 18907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:07.349973 18907 hybrid_clock.cc:648] HybridClock initialized: now 1785984187349952 us; error 63 us; skew 500 ppm
Aug 06 02:43:07 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:07.352726 18907 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:43:07.353628 18907 webserver.cc:460] Webserver started at http://127.28.191.2:40597/ using document root <none> and password file <none>
I20260806 02:43:07.354188 18907 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:07.354249 18907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:07.357945 18907 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:07.359933 18940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.361224 18907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:07.361344 18907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "1e7f7184e5f145ce9c39f7febc13fdad"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "de60135ccf99aff78a444d73fea73c45"
server_key_iv: "a975fa87570c5b765d1b0db28465052b"
server_key_version: "encryptionkey@0"
I20260806 02:43:07.361877 18907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:07.374305 18907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:07.377736 18907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:07.377946 18907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:07.378564 18907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:07.379402 18907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:07.379457 18907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.379523 18907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:07.379573 18907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.388832 18907 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:32909
I20260806 02:43:07.388865 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:32909 every 8 connection(s)
I20260806 02:43:07.389952 18907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:43:07.390990 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 18907
I20260806 02:43:07.391502 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:37139
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:45297
--webserver_port=35765
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:07.394209 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.392295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:58663)
0806 02:43:07.392792 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.393387 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
W20260806 02:43:07.394795 19095 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:45297: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
I20260806 02:43:07.395468 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.395248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:57465)
0806 02:43:07.395365 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.395413 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:43:07.396171 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.395977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:45965)
0806 02:43:07.396080 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.396121 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:43:07.396404 19095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:43:07.507160 19099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:07.507428 19099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:07.507508 19099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:07.511493 19099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:07.511574 19099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:07.511662 19099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:43:07.516714 19099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:37139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=35765
--tserver_master_addrs=127.28.191.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:07.517771 19099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:07.518778 19099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:07.528318 19108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:07.528271 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:07.528275 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:07.528719 19099 server_base.cc:1034] running on GCE node
I20260806 02:43:07.529219 19099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:07.529842 19099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:07.531031 19099 hybrid_clock.cc:648] HybridClock initialized: now 1785984187531015 us; error 36 us; skew 500 ppm
Aug 06 02:43:07 dist-test-slave-tfpf krb5kdc[17031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:07.533726 19099 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:43:07.534572 19099 webserver.cc:460] Webserver started at http://127.28.191.3:35765/ using document root <none> and password file <none>
I20260806 02:43:07.535133 19099 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:07.535225 19099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:07.539039 19099 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 02:43:07.541455 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.542721 19099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:07.542883 19099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "6ca7fdb736474ca98898b01b36fa7853"
format_stamp: "Formatted at 2026-08-06 02:43:05 on dist-test-slave-tfpf"
server_key: "79dc8b49cc40a29642414c82f282bafa"
server_key_iv: "993c0116a1dc04e29d4e4db4fde57b1b"
server_key_version: "encryptionkey@0"
I20260806 02:43:07.543465 19099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:07.559562 19099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:07.562590 19099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:07.562783 19099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:07.563418 19099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:07.564251 19099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:07.564322 19099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.564386 19099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:07.564438 19099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:07.573557 19099 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:37139
I20260806 02:43:07.573581 19284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:37139 every 8 connection(s)
I20260806 02:43:07.574537 19099 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:43:07.578583 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.576890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:35153)
0806 02:43:07.577395 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.577845 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":163,"threads_started":1}
W20260806 02:43:07.579090 19285 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:45297 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:45297: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
I20260806 02:43:07.579663 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.579456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:50049)
0806 02:43:07.579566 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.579612 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 02:43:07.580075 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 19099
I20260806 02:43:07.580322 19287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:07.580114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:36625)
0806 02:43:07.580221 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:07.580274 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:43:07.580503 19285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:43:08.281807 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.281332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:54267)
0806 02:43:08.281642 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:08.281703 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260806 02:43:08.397354 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.396923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:40987)
0806 02:43:08.397201 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:08.397252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260806 02:43:08.427083 19291 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:08.427313 19291 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:08.428418 19291 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
Aug 06 02:43:08 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.2@KRBTEST.COM
Aug 06 02:43:08 dist-test-slave-tfpf krb5kdc[17031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984187, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.3@KRBTEST.COM
I20260806 02:43:08.441924 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.428649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:32909 (local address 127.28.191.1:55935)
0806 02:43:08.428950 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:08.428964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:08.429104 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:08.430539 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:08.430545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:08.431100 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:08.431507 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:08.431520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.432783 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.432786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:08.433778 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:08.433788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.434474 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.434477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:08.434647 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:08.435378 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:08.435394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:08.437001 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:08.439435 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:08.439441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:08.439453 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:08.439711 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:08.440124 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:08.440128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:08.440130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:08.440263 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:08.440892 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:08.440898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:08.441194 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 02:43:08.441371 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 02:43:08.441654 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":124,"thread_start_us":58,"threads_started":1}
I20260806 02:43:08.441941 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.428845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:55935 (local address 127.28.191.2:32909)
0806 02:43:08.429231 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:08.429251 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:08.429281 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:43:08.429898 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:08.429903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:08.430201 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:08.430492 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:08.431684 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.432648 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.433899 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.434350 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.434524 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:08.437260 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:08.437265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:08.437275 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:08.437327 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:08.439299 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:08.439878 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:08.439885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:08.439895 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:08.440026 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:08.440369 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:08.440372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:08.440374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:08.440764 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:08.440914 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:08.441495 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:43:08.441759 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260806 02:43:08.441946 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.428592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37139 (local address 127.28.191.1:42529)
0806 02:43:08.428786 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:08.428800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:08.428901 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:08.430540 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:08.430546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:08.431269 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:08.431508 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:08.431521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.432852 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.432858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:08.433780 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:08.433791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.434539 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.434544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:08.434650 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:08.435444 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:08.435457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:08.437444 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:08.439896 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:08.439899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:08.439901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:08.440136 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:08.440552 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:08.440556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:08.440558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:08.440653 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:08.441346 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:08.441349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:08.441462 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 02:43:08.441588 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:43:08.441738 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":96}
I20260806 02:43:08.442837 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.428765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:42529 (local address 127.28.191.3:37139)
0806 02:43:08.429190 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:08.429202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:08.429242 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:43:08.429880 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:08.429885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:08.430206 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:08.430493 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:08.431703 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.432701 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.433921 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:08.434395 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:08.434588 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:08.437721 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:08.437728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:08.437743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:08.437788 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:08.439746 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:08.440256 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:08.440262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:08.440268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:08.440418 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:08.440754 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:08.440759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:08.440761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:08.441218 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:08.441363 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:08.441736 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 02:43:08.442694 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
W20260806 02:43:08.444384 18747 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
W20260806 02:43:08.444779 18745 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:08.444850 18745 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb6443500a749e18ad8b4fe2de7cf82; no voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853
I20260806 02:43:08.445122 19291 raft_consensus.cc:2726] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:43:08.581393 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:08.580989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:42079)
0806 02:43:08.581255 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:08.581303 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260806 02:43:09.283026 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:09.282620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:54929)
0806 02:43:09.282872 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:09.282929 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260806 02:43:09.398434 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:09.398036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:53281)
0806 02:43:09.398290 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:09.398346 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260806 02:43:09.582504 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:09.582097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:43153)
0806 02:43:09.582341 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:09.582396 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260806 02:43:10.284161 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:10.283757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:38657)
0806 02:43:10.283989 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:10.284048 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260806 02:43:10.366082 19304 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:10.366305 19304 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:10.366770 19304 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
W20260806 02:43:10.367638 18747 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
W20260806 02:43:10.367774 18745 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:10.367871 18745 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb6443500a749e18ad8b4fe2de7cf82; no voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853
I20260806 02:43:10.368038 19304 raft_consensus.cc:2726] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:43:10.399627 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:10.399203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:39939)
0806 02:43:10.399458 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:10.399512 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260806 02:43:10.583653 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:10.583267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:58069)
0806 02:43:10.583516 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:10.583566 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260806 02:43:11.285429 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:11.284962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:56045)
0806 02:43:11.285272 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:11.285330 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260806 02:43:11.400842 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:11.400337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:49281)
0806 02:43:11.400678 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:11.400735 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260806 02:43:11.584904 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:11.584500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:56757)
0806 02:43:11.584760 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:11.584813 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260806 02:43:12.286763 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:12.286331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:50753)
0806 02:43:12.286612 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:12.286668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
I20260806 02:43:12.402001 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:12.401600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:42007)
0806 02:43:12.401845 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:12.401903 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260806 02:43:12.586051 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:12.585622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:54049)
0806 02:43:12.585900 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:12.585953 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260806 02:43:12.735334 19319 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:12.735478 19319 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:12.735942 19319 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
W20260806 02:43:12.736733 18745 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
W20260806 02:43:12.736879 18747 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:12.736967 18747 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb6443500a749e18ad8b4fe2de7cf82; no voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853
I20260806 02:43:12.737172 19319 raft_consensus.cc:2726] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:43:13.287917 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:13.287492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:57191)
0806 02:43:13.287768 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:13.287824 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260806 02:43:13.403095 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:13.402699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:41163)
0806 02:43:13.402922 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:13.402971 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260806 02:43:13.587261 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:13.586831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:37735)
0806 02:43:13.587111 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:13.587166 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260806 02:43:14.289181 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:14.288703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:47041)
0806 02:43:14.288964 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:14.289021 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260806 02:43:14.404309 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:14.403810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:59915)
0806 02:43:14.404157 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:14.404215 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260806 02:43:14.588526 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:14.587996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:46289)
0806 02:43:14.588380 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:14.588434 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":75,"threads_started":1}
I20260806 02:43:15.290443 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:15.289998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:59585)
0806 02:43:15.290286 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:15.290343 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260806 02:43:15.401455 19328 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:15.401597 19328 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:15.402083 19328 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
W20260806 02:43:15.402890 18745 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
W20260806 02:43:15.403036 18747 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:15.403178 18747 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb6443500a749e18ad8b4fe2de7cf82; no voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853
I20260806 02:43:15.403345 19328 raft_consensus.cc:2726] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:43:15.405400 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:15.404947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:39215)
0806 02:43:15.405254 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:15.405313 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":135,"threads_started":1}
I20260806 02:43:15.589865 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:15.589440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:59821)
0806 02:43:15.589715 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:15.589771 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260806 02:43:16.291877 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:16.291386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:43981)
0806 02:43:16.291730 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:16.291781 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260806 02:43:16.406458 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:16.406089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:55005)
0806 02:43:16.406308 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:16.406371 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260806 02:43:16.591048 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:16.590649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:58709)
0806 02:43:16.590897 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:16.590952 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260806 02:43:17.293555 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:17.292972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:33671)
0806 02:43:17.293375 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:17.293440 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":192,"threads_started":1}
I20260806 02:43:17.407663 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:17.407267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:48263)
0806 02:43:17.407507 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:17.407565 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260806 02:43:17.592583 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:17.592083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:60155)
0806 02:43:17.592403 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:17.592476 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260806 02:43:18.231928 19341 raft_consensus.cc:482] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:18.232177 19341 raft_consensus.cc:504] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca7fdb736474ca98898b01b36fa7853" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37139 } } peers { permanent_uuid: "1e7f7184e5f145ce9c39f7febc13fdad" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 32909 } } peers { permanent_uuid: "7fb6443500a749e18ad8b4fe2de7cf82" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 40593 } }
I20260806 02:43:18.232704 19341 leader_election.cc:288] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139), 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909)
W20260806 02:43:18.233667 18745 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6ca7fdb736474ca98898b01b36fa7853 (127.28.191.3:37139): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
W20260806 02:43:18.233840 18747 leader_election.cc:341] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e7f7184e5f145ce9c39f7febc13fdad (127.28.191.2:32909): Not found: Tablet not found: 936332885ca540138dfcc7d6a26c0b09
I20260806 02:43:18.233939 18747 leader_election.cc:302] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb6443500a749e18ad8b4fe2de7cf82; no voters: 1e7f7184e5f145ce9c39f7febc13fdad, 6ca7fdb736474ca98898b01b36fa7853
I20260806 02:43:18.234138 19341 raft_consensus.cc:2726] T 936332885ca540138dfcc7d6a26c0b09 P 7fb6443500a749e18ad8b4fe2de7cf82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:43:18.295048 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:18.294595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:47219)
0806 02:43:18.294877 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:18.294941 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260806 02:43:18.408861 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:18.408436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:60215)
0806 02:43:18.408705 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:18.408764 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260806 02:43:18.593976 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:18.593535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:44847)
0806 02:43:18.593808 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:18.593874 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260806 02:43:19.296350 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:19.295881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:38079)
0806 02:43:19.296181 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:19.296251 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":152,"threads_started":1}
I20260806 02:43:19.410166 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:19.409722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:36105)
0806 02:43:19.409994 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:19.410067 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260806 02:43:19.595180 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:19.594728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:49703)
0806 02:43:19.595010 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:19.595064 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20260806 02:43:20.297762 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:20.297303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:44819)
0806 02:43:20.297601 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:20.297663 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260806 02:43:20.411360 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:20.410932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:46685)
0806 02:43:20.411216 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:20.411268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":129,"threads_started":1}
I20260806 02:43:20.596401 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:20.595965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:55053)
0806 02:43:20.596254 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:20.596306 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260806 02:43:21.299084 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:21.298615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:34303)
0806 02:43:21.298907 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:21.298970 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260806 02:43:21.412434 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:21.412054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:56389)
0806 02:43:21.412291 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:21.412343 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260806 02:43:21.597630 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:21.597206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:45571)
0806 02:43:21.597468 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:21.597523 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260806 02:43:22.300374 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:22.299983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.1:59171)
0806 02:43:22.300225 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:22.300278 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260806 02:43:22.413913 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:22.413417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.2:38313)
0806 02:43:22.413708 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:22.413789 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":130,"threads_started":1}
I20260806 02:43:22.585786 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 18676
I20260806 02:43:22.599059 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:22.598608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:45297 (local address 127.28.191.3:52665)
0806 02:43:22.598875 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:22.598939 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260806 02:43:22.599888 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 18907
I20260806 02:43:22.609402 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 19099
2026-08-06T02:43:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19402 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:43:22 dist-test-slave-tfpf krb5kdc[19367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:43:22 dist-test-slave-tfpf krb5kdc[19367](info): set up 2 sockets
Aug 06 02:43:22 dist-test-slave-tfpf krb5kdc[19367](info): commencing operation
krb5kdc: starting...
W20260806 02:43:23.836747 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:43:23 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:43:23.963733 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:24.076161 19383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:24.076443 19383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:24.076521 19383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:24.080451 19383 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:43:24.080523 19383 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:43:24.080550 19383 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:43:24.080585 19383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:24.080624 19383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:24.080669 19383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:24.080720 19383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:24.080760 19383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:24.085801 19383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:24.086740 19383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:24.087693 19383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:24.096232 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.096259 19391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.096272 19383 server_base.cc:1034] running on GCE node
W20260806 02:43:24.096292 19394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.096920 19383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:24.098243 19383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:24.099428 19383 hybrid_clock.cc:648] HybridClock initialized: now 1785984204099398 us; error 42 us; skew 500 ppm
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:24.107044 19383 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:24.107963 19383 webserver.cc:460] Webserver started at http://127.28.191.62:42401/ using document root <none> and password file <none>
I20260806 02:43:24.108531 19383 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:24.108613 19383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:24.108827 19383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:24.110962 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "3f350a0295a44ffaaa7a171b9db8dfb6"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "b0b0303bb2415871ab75c64e9563d5dc"
server_key_iv: "a3ba1c5d55fdde4279ebc57abddfabd9"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.111543 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "3f350a0295a44ffaaa7a171b9db8dfb6"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "b0b0303bb2415871ab75c64e9563d5dc"
server_key_iv: "a3ba1c5d55fdde4279ebc57abddfabd9"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.115350 19383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:43:24.117801 19404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.119005 19383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:24.119163 19383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "3f350a0295a44ffaaa7a171b9db8dfb6"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "b0b0303bb2415871ab75c64e9563d5dc"
server_key_iv: "a3ba1c5d55fdde4279ebc57abddfabd9"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.119278 19383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:24.132848 19383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:24.135730 19383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:24.135949 19383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:24.143860 19383 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:34267
I20260806 02:43:24.144153 19479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:34267 every 8 connection(s)
I20260806 02:43:24.144876 19383 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:24.147689 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:24.150157 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 19383
I20260806 02:43:24.150291 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:43:24.150620 29436 external_mini_cluster.cc:1428] Setting key 9a9a1a11986b725b815fec64bf49fff6
I20260806 02:43:24.153559 19480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap starting.
I20260806 02:43:24.155695 19480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:24.156423 19480 log.cc:824] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:24.158397 19480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: No bootstrap required, opened a new log
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:24.162582 19480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:24.162860 19480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:24.162933 19480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Initialized, Role: FOLLOWER
I20260806 02:43:24.163386 19480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:24.163513 19480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:24.163595 19480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:24.163697 19480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:24.164609 19480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:24.164949 19480 leader_election.cc:302] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f350a0295a44ffaaa7a171b9db8dfb6; no voters:
I20260806 02:43:24.165297 19480 leader_election.cc:288] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:43:24.165365 19488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:24.165498 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.152027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.28.191.62:34267)
0806 02:43:24.152458 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.152475 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.152514 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:43:24.153238 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.153258 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.153576 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.153943 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:24.154832 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.155858 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.156808 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.157359 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.157552 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.160445 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:24.160450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:24.160462 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:24.160487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:24.162271 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.162782 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.162787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.162791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.162975 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.163293 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.163297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.163299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.163814 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:24.164014 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.164329 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.164614 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260806 02:43:24.165599 19488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 LEADER]: Becoming Leader. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Running, Role: LEADER
I20260806 02:43:24.166574 19480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:24.167003 19488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:24.169099 19490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f350a0295a44ffaaa7a171b9db8dfb6. Latest consensus state: current_term: 1 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:43:24.169145 19489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:43:24.169226 19490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:24.169255 19489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
W20260806 02:43:24.171267 19506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:43:24.171355 19506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:43:24.171434 19508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:24.172111 19508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:24.176817 19508 catalog_manager.cc:1347] Generated new cluster ID: bbe874944b6640cc86e658f1a1d7459d
I20260806 02:43:24.176880 19508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:43:24.190699 19508 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:43:24.191592 19508 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:24.197642 19508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Generated new TSK 0
I20260806 02:43:24.198247 19508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:43:24.230206 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:24.346091 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:24.346323 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:24.346364 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:24.350252 19517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:24.350304 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:24.350360 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:43:24.355336 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:24.356308 19517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:24.357327 19517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:24.367663 19529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.367667 19526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.367846 19517 server_base.cc:1034] running on GCE node
W20260806 02:43:24.367910 19527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.368472 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:24.369138 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:24.370308 19517 hybrid_clock.cc:648] HybridClock initialized: now 1785984204370286 us; error 32 us; skew 500 ppm
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:24.373306 19517 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:43:24.374150 19517 webserver.cc:460] Webserver started at http://127.28.191.1:37491/ using document root <none> and password file <none>
I20260806 02:43:24.374680 19517 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:24.374739 19517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:24.374910 19517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:24.376991 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "8ebcb512d345431c9499f02e48a1048f"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "72a2f32712424f1e02e6856a3a757704"
server_key_iv: "dd75c72c27baacb1954638a7c4eac883"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.377604 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "8ebcb512d345431c9499f02e48a1048f"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "72a2f32712424f1e02e6856a3a757704"
server_key_iv: "dd75c72c27baacb1954638a7c4eac883"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.381253 19517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:43:24.383594 19540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.384801 19517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:24.384938 19517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "8ebcb512d345431c9499f02e48a1048f"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "72a2f32712424f1e02e6856a3a757704"
server_key_iv: "dd75c72c27baacb1954638a7c4eac883"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.385051 19517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:24.422451 19517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:24.425961 19517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:24.426173 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:24.426783 19517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:24.427634 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:24.427703 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.427773 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:24.427819 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.436820 19517 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:32895
I20260806 02:43:24.436843 19698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:32895 every 8 connection(s)
I20260806 02:43:24.437912 19517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:24.447449 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 19517
I20260806 02:43:24.447602 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:43:24.447885 29436 external_mini_cluster.cc:1428] Setting key 5888d90d3868653428ccaf40105f5d2e
I20260806 02:43:24.451748 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.439738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:41935 (local address 127.28.191.62:34267)
0806 02:43:24.439877 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.439881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.440702 (+ 821us) server_negotiation.cc:407] Connection header received
0806 02:43:24.442018 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.442022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.442096 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.442197 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:24.443418 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.443982 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.445174 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.445421 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.445535 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.448032 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:24.448036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:24.448038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:24.448064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:24.449701 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.450238 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.450242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.450244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.450324 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.450669 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.450674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.450676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.450876 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:24.450985 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.451468 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.451591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:43:24.452548 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.440072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:41935)
0806 02:43:24.440526 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.440579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.441795 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.442378 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.442384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.442759 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:24.443246 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.443260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.444120 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.444129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.444994 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.445004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.445588 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.445592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.445759 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.446314 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:24.446330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:24.447852 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:24.449818 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.449824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.449839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.450134 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.450416 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.450420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.450422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.450565 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.450986 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:24.450994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:24.451238 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.451427 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.451768 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260806 02:43:24.453768 19699 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:24.454053 19699 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:24.454698 19699 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:24.456559 19427 ts_manager.cc:194] Registered new tserver with Master: 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895)
I20260806 02:43:24.458210 19427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:41935
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:43:24.477984 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:24.590266 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:24.590504 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:24.590552 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:24.594528 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:24.594587 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:24.594668 19706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:43:24.599759 19706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:24.600790 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:24.601835 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:24.611362 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.611423 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.611354 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.611672 19706 server_base.cc:1034] running on GCE node
I20260806 02:43:24.612020 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:24.612566 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:24.613740 19706 hybrid_clock.cc:648] HybridClock initialized: now 1785984204613717 us; error 46 us; skew 500 ppm
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:24.616446 19706 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:43:24.617437 19706 webserver.cc:460] Webserver started at http://127.28.191.2:42933/ using document root <none> and password file <none>
I20260806 02:43:24.617983 19706 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:24.618072 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:24.618283 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:24.620273 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "eaae90fdedcb46ce9f3b295c682b9216"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "574a801461588c3d2fbb4daace1e9229"
server_key_iv: "8f0ae35927ccd50d4e5508ad6f8be99a"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.620838 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "eaae90fdedcb46ce9f3b295c682b9216"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "574a801461588c3d2fbb4daace1e9229"
server_key_iv: "8f0ae35927ccd50d4e5508ad6f8be99a"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.624725 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:43:24.627276 19728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.628696 19706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:43:24.628847 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "eaae90fdedcb46ce9f3b295c682b9216"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "574a801461588c3d2fbb4daace1e9229"
server_key_iv: "8f0ae35927ccd50d4e5508ad6f8be99a"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.628973 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:24.641373 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:24.644440 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:24.644656 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:24.645267 19706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:24.646090 19706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:24.646158 19706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.646245 19706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:24.646286 19706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.654842 19706 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:40539
I20260806 02:43:24.654870 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:40539 every 8 connection(s)
I20260806 02:43:24.655730 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:43:24.664722 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 19706
I20260806 02:43:24.664860 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:24.665237 29436 external_mini_cluster.cc:1428] Setting key 7d60aa3e4b72a61705916780e434b803
I20260806 02:43:24.670115 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.657636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:60575 (local address 127.28.191.62:34267)
0806 02:43:24.657786 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.657791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.658656 (+ 865us) server_negotiation.cc:407] Connection header received
0806 02:43:24.659837 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.659840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.659889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.659966 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:24.661078 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.661605 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.662710 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.662927 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.663018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.665649 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:24.665653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:24.665656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:24.665683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:24.667272 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.667970 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.667976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.667981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.668075 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.668565 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.668571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.668575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.668833 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:24.668918 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.669882 (+ 964us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.669998 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:43:24.671185 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.658016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:60575)
0806 02:43:24.658491 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.658533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.659649 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.660125 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.660131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.660472 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:24.660935 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.660947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.661723 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.661727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.662602 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.662612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.663137 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.663141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.663278 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.663786 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:24.663801 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:24.665493 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:24.667417 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.667425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.667440 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.667845 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.668194 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.668200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.668204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.668434 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.669015 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:24.669024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:24.669542 (+ 518us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.669816 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.670175 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260806 02:43:24.672472 19895 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:24.672729 19895 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:24.673300 19895 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:24.674427 19427 ts_manager.cc:194] Registered new tserver with Master: eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:24.675242 19427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:60575
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:43:24.695844 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:24.810115 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:24.810359 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:24.810437 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:24.814519 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:24.814589 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:24.814695 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:43:24.820065 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:24.821214 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:24.822252 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:24.832368 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.832377 19915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:24.832377 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:24.832434 19906 server_base.cc:1034] running on GCE node
I20260806 02:43:24.833050 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:24.833714 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:24.834870 19906 hybrid_clock.cc:648] HybridClock initialized: now 1785984204834829 us; error 58 us; skew 500 ppm
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:24.837716 19906 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:43:24.838587 19906 webserver.cc:460] Webserver started at http://127.28.191.3:34653/ using document root <none> and password file <none>
I20260806 02:43:24.839165 19906 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:24.839229 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:24.839470 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:43:24.841533 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "f4cbf6a7d5666524be13be18ebb03542"
server_key_iv: "1d7b30c3b1c94bb180d705e0eef843bb"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.842170 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "f4cbf6a7d5666524be13be18ebb03542"
server_key_iv: "1d7b30c3b1c94bb180d705e0eef843bb"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.845850 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:43:24.848248 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.849493 19906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:24.849608 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "f4cbf6a7d5666524be13be18ebb03542"
server_key_iv: "1d7b30c3b1c94bb180d705e0eef843bb"
server_key_version: "encryptionkey@0"
I20260806 02:43:24.849711 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:24.864735 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:24.867707 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:24.867924 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:24.868474 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:24.869324 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:24.869381 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.869431 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:24.869446 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:24.878028 19906 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:33951
I20260806 02:43:24.878091 20080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:33951 every 8 connection(s)
I20260806 02:43:24.878925 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:43:24.882807 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 19906
I20260806 02:43:24.882974 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:43:24.883332 29436 external_mini_cluster.cc:1428] Setting key dee1dc8dff4c4f0e94399432c19a1f68
Aug 06 02:43:24 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984204, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:24.892196 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.880681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:45335 (local address 127.28.191.62:34267)
0806 02:43:24.880823 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.880827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.881677 (+ 850us) server_negotiation.cc:407] Connection header received
0806 02:43:24.883032 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.883036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.883094 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.883205 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:24.884509 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.885289 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.886391 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.886624 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.886727 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.889021 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:24.889024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:24.889026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:24.889053 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:24.890507 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.890967 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.890969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.890970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.891047 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.891330 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.891332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.891333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.891494 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:24.891581 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.891986 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.892082 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:43:24.892944 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.881050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:45335)
0806 02:43:24.881526 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.881572 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.882756 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.883376 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.883383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.883790 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:24.884347 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.884361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.885404 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.885408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.886264 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.886274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.886775 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.886779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.886940 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.887454 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:24.887470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:24.888868 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:24.890626 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.890630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.890641 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.890883 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.891127 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:24.891129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:24.891130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:24.891253 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:24.891578 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:24.891583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:24.891792 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.891941 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.892186 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260806 02:43:24.894084 20081 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:24.894284 20081 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:24.894701 20081 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:24.895653 19427 ts_manager.cc:194] Registered new tserver with Master: 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951)
I20260806 02:43:24.896396 19427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:45335
I20260806 02:43:24.897133 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:43:24.906136 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.899100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.28.191.62:34267)
0806 02:43:24.899230 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.899234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.899366 (+ 132us) server_negotiation.cc:407] Connection header received
0806 02:43:24.899451 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.899453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.899487 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.899558 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:24.900205 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.900668 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.901487 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.901692 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.901769 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.903163 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:24.903166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:24.903167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:24.903189 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:24.904571 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.904948 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.904951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.904952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.905023 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:24.905338 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:24.905341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:24.905342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:24.905536 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:24.905657 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.905870 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.905955 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:43:24.908941 19427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:43:24.911935 19427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:43:24.927059 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.920630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.28.191.1:32895)
0806 02:43:24.921763 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.921781 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.921804 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:43:24.921883 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.921891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.922177 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.922339 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:24.922838 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.923865 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.925462 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.926317 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926449 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.926615 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.926796 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.926891 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":105,"threads_started":1}
I20260806 02:43:24.927091 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.920478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:32895 (local address 127.0.0.1:52534)
0806 02:43:24.921267 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.921320 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.921457 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.922339 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.922345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.922367 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:24.922531 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.922539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.924035 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.924039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.925341 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.925352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.926456 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.926576 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.926628 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.926764 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.926899 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":115,"thread_start_us":79,"threads_started":1}
I20260806 02:43:24.927645 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.921458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.28.191.2:40539)
0806 02:43:24.922000 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.922014 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.922040 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:43:24.922125 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.922130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.922348 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.922558 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:24.922806 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.924014 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.925625 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.926388 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926990 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.927151 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.927404 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.927514 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":112,"threads_started":1}
I20260806 02:43:24.927831 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.920955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:33951 (local address 127.0.0.1:43214)
0806 02:43:24.922229 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.922245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.922346 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.922900 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.922903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.922922 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:24.923007 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.923013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.924330 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.924333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.925938 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.925947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.927112 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.927363 (+ 251us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.927466 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.927508 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.927592 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.927672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":84,"thread_start_us":132,"threads_started":1}
I20260806 02:43:24.928262 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.921752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.28.191.3:33951)
0806 02:43:24.922173 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:24.922185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:24.922456 (+ 271us) server_negotiation.cc:407] Connection header received
0806 02:43:24.922546 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:24.922552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:24.922736 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:24.922925 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:24.923098 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.924178 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926068 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.926792 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926899 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.927041 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:24.928030 (+ 989us) server_negotiation.cc:299] Negotiation successful
0806 02:43:24.928143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":174,"threads_started":1}
I20260806 02:43:24.928083 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:24.920684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.0.0.1:59946)
0806 02:43:24.921329 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:24.921354 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:24.921462 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:24.922550 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:24.922553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:24.922574 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:24.922684 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.922690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.924182 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.924187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.925468 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:24.925491 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:24.926533 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:24.926537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:24.926627 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:24.926655 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:43:24.926718 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:43:24.926871 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":150,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260806 02:43:24.930302 19604 tablet_service.cc:1467] Processing CreateTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 (DEFAULT_TABLE table=test-table [id=b4de76da34e54030bd31f2d97795f030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:24.931485 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eba739aa5cb47399e953e3bbf8c9167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:24.931164 19802 tablet_service.cc:1467] Processing CreateTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 (DEFAULT_TABLE table=test-table [id=b4de76da34e54030bd31f2d97795f030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:24.931247 19994 tablet_service.cc:1467] Processing CreateTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 (DEFAULT_TABLE table=test-table [id=b4de76da34e54030bd31f2d97795f030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:24.932145 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eba739aa5cb47399e953e3bbf8c9167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:24.932360 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eba739aa5cb47399e953e3bbf8c9167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:24.938609 20100 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Bootstrap starting.
I20260806 02:43:24.939837 20101 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap starting.
I20260806 02:43:24.941286 20100 tablet_bootstrap.cc:654] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:24.941388 20102 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Bootstrap starting.
I20260806 02:43:24.941625 20101 tablet_bootstrap.cc:654] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:24.942440 20100 log.cc:824] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:24.942883 20101 log.cc:824] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:24.943746 20102 tablet_bootstrap.cc:654] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:24.944343 20100 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: No bootstrap required, opened a new log
I20260806 02:43:24.944388 20102 log.cc:824] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:24.944583 20100 ts_tablet_manager.cc:1397] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:43:24.946211 20102 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: No bootstrap required, opened a new log
I20260806 02:43:24.946389 20102 ts_tablet_manager.cc:1397] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:43:24.946703 20101 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: No bootstrap required, opened a new log
I20260806 02:43:24.946877 20101 ts_tablet_manager.cc:1397] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:43:24.947825 20100 raft_consensus.cc:348] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.948078 20100 raft_consensus.cc:374] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:24.948161 20100 raft_consensus.cc:729] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265b8a5f5f904a308e7e3d34b22dd9fd, State: Initialized, Role: FOLLOWER
I20260806 02:43:24.948727 20100 consensus_queue.cc:260] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.949717 20100 ts_tablet_manager.cc:1428] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:43:24.950114 20081 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
I20260806 02:43:24.950160 20102 raft_consensus.cc:348] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.950423 20102 raft_consensus.cc:374] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:24.950474 20102 raft_consensus.cc:729] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaae90fdedcb46ce9f3b295c682b9216, State: Initialized, Role: FOLLOWER
I20260806 02:43:24.950555 20101 raft_consensus.cc:348] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.950783 20101 raft_consensus.cc:374] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:24.950886 20101 raft_consensus.cc:729] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebcb512d345431c9499f02e48a1048f, State: Initialized, Role: FOLLOWER
I20260806 02:43:24.950891 20102 consensus_queue.cc:260] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.951428 20101 consensus_queue.cc:260] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:24.951946 20102 ts_tablet_manager.cc:1428] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:43:24.952230 19699 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
I20260806 02:43:24.952316 19895 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
I20260806 02:43:24.952453 20101 ts_tablet_manager.cc:1428] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260806 02:43:25.130064 20082 tablet.cc:2369] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:43:25.156935 19898 tablet.cc:2369] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:43:25.189278 19701 tablet.cc:2369] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:43:25.212131 20106 raft_consensus.cc:482] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:25.212361 20106 raft_consensus.cc:504] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:25.213537 20106 leader_election.cc:288] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:25.217650 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.213788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.28.191.3:47775)
0806 02:43:25.214213 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:25.214227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:25.214335 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:25.214587 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:25.214589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:25.214601 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:25.214691 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:25.214700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.215501 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.215504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:25.216491 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:25.216501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.217303 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.217305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:25.217382 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:25.217403 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:43:25.217464 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:43:25.217518 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":55,"thread_start_us":137,"threads_started":1}
I20260806 02:43:25.217716 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.213917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:47775 (local address 127.28.191.2:40539)
0806 02:43:25.214045 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:25.214050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:25.214351 (+ 301us) server_negotiation.cc:407] Connection header received
0806 02:43:25.214412 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:25.214418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:25.214479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:25.214564 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:25.214814 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.215400 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.216640 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.217044 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.217321 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:25.217377 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:25.217517 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:43:25.217593 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:43:25.218397 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.213788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:32895 (local address 127.28.191.3:49421)
0806 02:43:25.214010 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:25.214025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:25.214130 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:25.214495 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:25.214497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:25.214510 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:25.214659 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:25.214668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.216146 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.216149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:25.217000 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:25.217009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.217986 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.217989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:25.218101 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:25.218122 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:43:25.218215 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:43:25.218288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260806 02:43:25.218493 19827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eba739aa5cb47399e953e3bbf8c9167" candidate_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaae90fdedcb46ce9f3b295c682b9216" is_pre_election: true
I20260806 02:43:25.218802 19827 raft_consensus.cc:2454] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265b8a5f5f904a308e7e3d34b22dd9fd in term 0.
I20260806 02:43:25.218847 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.213905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:49421 (local address 127.28.191.1:32895)
0806 02:43:25.214052 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:25.214057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:25.214075 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:43:25.214293 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:25.214296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:25.214386 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:25.214481 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:25.215541 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.216037 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.217277 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.217873 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.218003 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:25.218051 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:25.218693 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 02:43:25.218759 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:43:25.219307 19934 leader_election.cc:302] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216; no voters:
I20260806 02:43:25.219409 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eba739aa5cb47399e953e3bbf8c9167" candidate_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebcb512d345431c9499f02e48a1048f" is_pre_election: true
I20260806 02:43:25.219565 20106 raft_consensus.cc:2781] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:43:25.219657 20106 raft_consensus.cc:482] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:43:25.219724 20106 raft_consensus.cc:3037] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:25.219739 19635 raft_consensus.cc:2454] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265b8a5f5f904a308e7e3d34b22dd9fd in term 0.
I20260806 02:43:25.220782 20106 raft_consensus.cc:504] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:25.221161 20106 leader_election.cc:288] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [CANDIDATE]: Term 1 election: Requested vote from peers 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:25.221496 19827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eba739aa5cb47399e953e3bbf8c9167" candidate_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaae90fdedcb46ce9f3b295c682b9216"
I20260806 02:43:25.221509 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eba739aa5cb47399e953e3bbf8c9167" candidate_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebcb512d345431c9499f02e48a1048f"
I20260806 02:43:25.221626 19635 raft_consensus.cc:3037] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:25.221626 19827 raft_consensus.cc:3037] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:25.222978 19635 raft_consensus.cc:2454] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265b8a5f5f904a308e7e3d34b22dd9fd in term 1.
I20260806 02:43:25.222978 19827 raft_consensus.cc:2454] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265b8a5f5f904a308e7e3d34b22dd9fd in term 1.
I20260806 02:43:25.223296 19934 leader_election.cc:302] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216; no voters:
I20260806 02:43:25.223497 20106 raft_consensus.cc:2781] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:25.223724 20106 raft_consensus.cc:686] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 LEADER]: Becoming Leader. State: Replica: 265b8a5f5f904a308e7e3d34b22dd9fd, State: Running, Role: LEADER
I20260806 02:43:25.224038 20106 consensus_queue.cc:237] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:25.227118 19427 catalog_manager.cc:5495] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3). New cstate: current_term: 1 leader_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } health_report { overall_health: HEALTHY } } }
I20260806 02:43:25.287765 20099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.284503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.28.191.3:33951)
0806 02:43:25.284656 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:25.284660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:25.284674 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:43:25.284755 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:25.284757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:25.284812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:25.284905 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:25.285173 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.285680 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.286515 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:25.286730 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:25.286812 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:25.287066 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:25.287466 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:43:25.287574 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:25.287604 (+ 30us) server_negotiation.cc:299] Negotiation successful
0806 02:43:25.287653 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:43:25.293814 19827 raft_consensus.cc:1261] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Refusing update from remote peer 265b8a5f5f904a308e7e3d34b22dd9fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:25.293790 19635 raft_consensus.cc:1261] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Refusing update from remote peer 265b8a5f5f904a308e7e3d34b22dd9fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:25.294562 20111 consensus_queue.cc:1035] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:25.294752 20106 consensus_queue.cc:1035] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:25.312851 19427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:25.313051 19427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:25.315567 19427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Sending DeleteTablet for 3 replicas of tablet 6eba739aa5cb47399e953e3bbf8c9167
I20260806 02:43:25.316321 19604 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:52534
I20260806 02:43:25.316356 19802 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:59946
I20260806 02:43:25.316543 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 19517
I20260806 02:43:25.327060 19994 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:43214
I20260806 02:43:25.327121 20132 tablet_replica.cc:330] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: stopping tablet replica
I20260806 02:43:25.327551 20133 tablet_replica.cc:330] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: stopping tablet replica
I20260806 02:43:25.327566 20132 raft_consensus.cc:2229] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:25.327865 20133 raft_consensus.cc:2229] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:43:25.328126 20132 raft_consensus.cc:2258] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:25.328226 20133 raft_consensus.cc:2258] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:25.329758 20133 ts_tablet_manager.cc:1905] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:25.329996 20132 ts_tablet_manager.cc:1905] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:25.332579 20133 ts_tablet_manager.cc:1918] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:25.332687 20133 log.cc:1196] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/6eba739aa5cb47399e953e3bbf8c9167
I20260806 02:43:25.333038 20133 ts_tablet_manager.cc:1939] T 6eba739aa5cb47399e953e3bbf8c9167 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting consensus metadata
I20260806 02:43:25.334041 20132 ts_tablet_manager.cc:1918] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:25.334208 20132 log.cc:1196] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/6eba739aa5cb47399e953e3bbf8c9167
I20260806 02:43:25.334549 19406 catalog_manager.cc:4841] TS 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): tablet 6eba739aa5cb47399e953e3bbf8c9167 (table test-table [id=b4de76da34e54030bd31f2d97795f030]) successfully deleted
I20260806 02:43:25.334610 20132 ts_tablet_manager.cc:1939] T 6eba739aa5cb47399e953e3bbf8c9167 P eaae90fdedcb46ce9f3b295c682b9216: Deleting consensus metadata
W20260806 02:43:25.335187 19405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:32895 (error 108)
I20260806 02:43:25.335464 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 19706
I20260806 02:43:25.341997 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.339966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:32895 (local address 127.0.0.1:52536)
0806 02:43:25.341798 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:25.341886 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:32895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1716}
W20260806 02:43:25.342216 19405 catalog_manager.cc:4584] TS 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6eba739aa5cb47399e953e3bbf8c9167: Network error: Client connection negotiation failed: client connection to 127.28.191.1:32895: connect: Connection refused (error 111)
I20260806 02:43:25.348495 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 19906
I20260806 02:43:25.352900 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:25.348695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.0.0.1:59950)
0806 02:43:25.352729 (+ 4034us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:25.352796 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.2:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3918}
I20260806 02:43:25.358886 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 19383
I20260806 02:43:45.372224 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--webserver_interface=127.28.191.62
--webserver_port=42401
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:43:45.494263 20137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:45.494585 20137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:45.494668 20137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:45.498972 20137 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:43:45.499058 20137 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:43:45.499085 20137 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:43:45.499123 20137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:43:45.499168 20137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:45.499230 20137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:45.499270 20137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:43:45.499305 20137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:43:45.504817 20137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:45.505985 20137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:45.507040 20137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:45.522876 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:45.522886 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:45.522957 20137 server_base.cc:1034] running on GCE node
W20260806 02:43:45.522883 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:45.523849 20137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:45.525276 20137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:45.526512 20137 hybrid_clock.cc:648] HybridClock initialized: now 1785984225526500 us; error 43 us; skew 500 ppm
Aug 06 02:43:45 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:45.529639 20137 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:43:45.530639 20137 webserver.cc:460] Webserver started at http://127.28.191.62:42401/ using document root <none> and password file <none>
I20260806 02:43:45.531275 20137 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:45.531399 20137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:45.535534 20137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:45.537977 20166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:45.539288 20137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:43:45.539453 20137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "3f350a0295a44ffaaa7a171b9db8dfb6"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "b0b0303bb2415871ab75c64e9563d5dc"
server_key_iv: "a3ba1c5d55fdde4279ebc57abddfabd9"
server_key_version: "encryptionkey@0"
I20260806 02:43:45.540099 20137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:45.563575 20137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:45.566918 20137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:45.567145 20137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:45.575250 20137 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:34267
I20260806 02:43:45.575551 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:34267 every 8 connection(s)
I20260806 02:43:45.576449 20137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:43:45.581034 20233 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:43:45.581481 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20137
I20260806 02:43:45.581938 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:32895
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=37491
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:45.582921 20233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap starting.
I20260806 02:43:45.607770 20233 log.cc:824] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:45.620529 20233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:45.621217 20233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap complete.
I20260806 02:43:45.627280 20233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:45.627876 20233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Initialized, Role: FOLLOWER
I20260806 02:43:45.628486 20233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:45.628657 20233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:43:45.628732 20233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:43:45.628846 20233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:43:45.631906 20233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:45.632347 20233 leader_election.cc:302] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f350a0295a44ffaaa7a171b9db8dfb6; no voters:
I20260806 02:43:45.632814 20233 leader_election.cc:288] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:43:45.632959 20238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:43:45.633294 20238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 LEADER]: Becoming Leader. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Running, Role: LEADER
I20260806 02:43:45.633639 20238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:43:45.634317 20233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:43:45.635126 20240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f350a0295a44ffaaa7a171b9db8dfb6. Latest consensus state: current_term: 2 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:43:45.635157 20239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:43:45.635279 20239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:45.635279 20240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:43:45.636971 20250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:43:45.642176 20250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:43:45.642854 20250 catalog_manager.cc:1259] Loaded cluster ID: bbe874944b6640cc86e658f1a1d7459d
I20260806 02:43:45.642936 20250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:43:45.645743 20250 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:43:45.646473 20250 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Loaded TSK: 0
I20260806 02:43:45.665488 20250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Generated new TSK 1
I20260806 02:43:45.665776 20250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:43:45.723394 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:45.723723 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:45.723811 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:45.727875 20236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:45.727977 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:45.728080 20236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:43:45.733660 20236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:32895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=37491
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:45.734781 20236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:45.735960 20236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:45.747174 20270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:45.747169 20271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:45.747529 20273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:45.748021 20236 server_base.cc:1034] running on GCE node
I20260806 02:43:45.748463 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:45.749185 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:45.750370 20236 hybrid_clock.cc:648] HybridClock initialized: now 1785984225750349 us; error 38 us; skew 500 ppm
Aug 06 02:43:45 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:45.753253 20236 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:43:45.754271 20236 webserver.cc:460] Webserver started at http://127.28.191.1:37491/ using document root <none> and password file <none>
I20260806 02:43:45.754856 20236 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:45.754935 20236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:45.758987 20236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:45.761247 20282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:45.762533 20236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:43:45.762669 20236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "8ebcb512d345431c9499f02e48a1048f"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "72a2f32712424f1e02e6856a3a757704"
server_key_iv: "dd75c72c27baacb1954638a7c4eac883"
server_key_version: "encryptionkey@0"
I20260806 02:43:45.763209 20236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:45.796010 20236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:45.799661 20236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:45.799881 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:45.800509 20236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:45.801725 20291 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:43:45.805953 20236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:43:45.806022 20236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 02:43:45.806099 20236 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:43:45.807782 20236 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:43:45.807828 20236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:43:45.808120 20291 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap starting.
I20260806 02:43:45.819016 20236 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:32895
I20260806 02:43:45.819249 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:32895 every 8 connection(s)
I20260806 02:43:45.820070 20236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:43:45.821611 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20236
I20260806 02:43:45.822276 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:40539
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=42933
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:43:45.827478 20291 log.cc:824] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:43:45 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:45.837610 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:45.822132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:59309 (local address 127.28.191.62:34267)
0806 02:43:45.822816 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:45.822837 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:45.823277 (+ 440us) server_negotiation.cc:407] Connection header received
0806 02:43:45.824400 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:45.824418 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:45.824835 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:45.825239 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:45.826653 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.827722 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.829198 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.829671 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.829850 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:45.832268 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:45.832278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:45.832289 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:45.832316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:45.834059 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:45.834695 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:45.834701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:45.834705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:45.834820 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:45.835185 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:45.835190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:45.835192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:45.835689 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:45.835848 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:45.836364 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 02:43:45.836715 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":117,"threads_started":1}
I20260806 02:43:45.837634 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:45.822155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:59309)
0806 02:43:45.823113 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:45.823159 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:45.823960 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:45.825321 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:45.825330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:45.825820 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:45.826455 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:45.826477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.827883 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.827887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:45.829022 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:45.829033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.829818 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.829822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:45.830001 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:45.830623 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:45.830640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:45.832064 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:45.834199 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:45.834205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:45.834217 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:45.834563 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:45.834955 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:45.834957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:45.834958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:45.835083 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:45.835830 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:45.835836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:45.836065 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 02:43:45.836249 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 02:43:45.836585 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":147,"threads_started":1}
I20260806 02:43:45.837703 20291 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:43:45.838125 20291 tablet_bootstrap.cc:492] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap complete.
I20260806 02:43:45.839589 20291 ts_tablet_manager.cc:1397] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260806 02:43:45.840206 20425 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:45.840440 20425 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:45.840927 20425 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:45.842576 20291 raft_consensus.cc:348] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:45.843103 20291 raft_consensus.cc:729] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebcb512d345431c9499f02e48a1048f, State: Initialized, Role: FOLLOWER
I20260806 02:43:45.843345 20186 ts_manager.cc:194] Registered new tserver with Master: 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895)
I20260806 02:43:45.843642 20291 consensus_queue.cc:260] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } }
I20260806 02:43:45.844673 20291 ts_tablet_manager.cc:1428] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 02:43:45.847110 20186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:59309
I20260806 02:43:45.849771 20425 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
Aug 06 02:43:45 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.1@KRBTEST.COM
I20260806 02:43:45.857548 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:45.846445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:32895 (local address 127.0.0.1:44886)
0806 02:43:45.847088 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:45.847140 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:45.847316 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:45.847843 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:45.847849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:45.848301 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:45.848437 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:45.848446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.849757 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.849760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:45.850729 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:45.850744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.851546 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.851550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:45.851657 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:45.852255 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:45.852270 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:45.853825 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:45.855835 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:45.855843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:45.855848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:45.856065 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:45.856393 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:45.856395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:45.856397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:45.856472 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:45.856975 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:45.856980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:45.857157 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 02:43:45.857290 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:43:45.857428 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":102,"threads_started":1}
I20260806 02:43:45.857723 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:45.846556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.28.191.1:32895)
0806 02:43:45.847139 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:45.847153 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:45.847284 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:43:45.847472 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:45.847478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:45.847672 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:45.847821 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:45.848573 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.849629 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.850878 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:45.851415 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:45.851523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:45.853961 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:45.853971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:45.853983 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:45.854009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:45.855712 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:45.856178 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:45.856185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:45.856186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:45.856297 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:45.856559 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:45.856562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:45.856564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:45.856862 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:45.856997 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:45.857400 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 02:43:45.857566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":119,"threads_started":1}
I20260806 02:43:45.858642 20343 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eba739aa5cb47399e953e3bbf8c9167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:44886
I20260806 02:43:45.859448 20437 tablet_replica.cc:330] stopping tablet replica
I20260806 02:43:45.859735 20437 raft_consensus.cc:2229] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:45.859974 20437 raft_consensus.cc:2258] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:45.860909 20437 ts_tablet_manager.cc:1905] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:45.864483 20437 ts_tablet_manager.cc:1918] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:45.864609 20437 log.cc:1196] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal/wals/6eba739aa5cb47399e953e3bbf8c9167
I20260806 02:43:45.864933 20437 ts_tablet_manager.cc:1939] T 6eba739aa5cb47399e953e3bbf8c9167 P 8ebcb512d345431c9499f02e48a1048f: Deleting consensus metadata
I20260806 02:43:45.866684 20167 catalog_manager.cc:4841] TS 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895): tablet 6eba739aa5cb47399e953e3bbf8c9167 (table test-table [id=b4de76da34e54030bd31f2d97795f030]) successfully deleted
W20260806 02:43:45.950446 20430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:45.950757 20430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:45.950829 20430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:45.955011 20430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:45.955077 20430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:45.955137 20430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:43:45.960330 20430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:40539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=42933
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:45.961504 20430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:45.962489 20430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:45.972447 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:45.972440 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:45.972440 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:45.972579 20430 server_base.cc:1034] running on GCE node
I20260806 02:43:45.973209 20430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:45.973805 20430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:45.974967 20430 hybrid_clock.cc:648] HybridClock initialized: now 1785984225974948 us; error 32 us; skew 500 ppm
Aug 06 02:43:45 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:45.977826 20430 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:43:45.978677 20430 webserver.cc:460] Webserver started at http://127.28.191.2:42933/ using document root <none> and password file <none>
I20260806 02:43:45.979254 20430 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:45.979352 20430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:45.983299 20430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:45.985816 20456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:45.987265 20430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:43:45.987421 20430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "eaae90fdedcb46ce9f3b295c682b9216"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "574a801461588c3d2fbb4daace1e9229"
server_key_iv: "8f0ae35927ccd50d4e5508ad6f8be99a"
server_key_version: "encryptionkey@0"
I20260806 02:43:45.988016 20430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:46.000880 20430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:46.004393 20430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:46.004642 20430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:46.005401 20430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:46.006337 20430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:46.006414 20430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:46.006487 20430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:46.006539 20430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:46.016338 20430 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:40539
I20260806 02:43:46.016366 20615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:40539 every 8 connection(s)
I20260806 02:43:46.017457 20430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:43:46.022136 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20430
I20260806 02:43:46.022612 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:33951
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=34653
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:43:46 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984225, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:46.032832 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.019444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:33935 (local address 127.28.191.62:34267)
0806 02:43:46.019591 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.019595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.020412 (+ 817us) server_negotiation.cc:407] Connection header received
0806 02:43:46.021806 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.021810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.021892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.021996 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:46.023357 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.023904 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.025084 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.025363 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.025487 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.028563 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:46.028566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:46.028568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:46.028593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:46.030370 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.031077 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.031080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.031081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.031176 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.031602 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.031604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.031605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.031784 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:46.031859 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.032589 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.032706 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:43:46.034325 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.019793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:33935)
0806 02:43:46.020257 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.020300 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.021556 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.022179 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.022186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.022599 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:46.023181 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.023195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.024040 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.024044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.024924 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.024935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.025529 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.025533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.025707 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.026331 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:46.026348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:46.028359 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:46.030528 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:46.030537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:46.030556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:46.030968 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:46.031279 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:46.031284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:46.031286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:46.031499 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:46.031899 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:46.031908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:46.032299 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.032554 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.032969 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260806 02:43:46.035682 20616 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:46.035945 20616 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:46.036530 20616 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:46.037743 20186 ts_manager.cc:194] Registered new tserver with Master: eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:46.038515 20186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:33935
W20260806 02:43:46.146664 20620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:43:46.146922 20620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:43:46.146965 20620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:43:46.150930 20620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:43:46.150991 20620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:43:46.151049 20620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:43:46.156370 20620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:33951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=34653
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:43:46.157480 20620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:43:46.158457 20620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:43:46.168607 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:43:46.168561 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:46.168816 20620 server_base.cc:1034] running on GCE node
W20260806 02:43:46.168567 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:43:46.169404 20620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:43:46.170046 20620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:43:46.171205 20620 hybrid_clock.cc:648] HybridClock initialized: now 1785984226171174 us; error 46 us; skew 500 ppm
Aug 06 02:43:46 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:43:46.174132 20620 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:43:46.175058 20620 webserver.cc:460] Webserver started at http://127.28.191.3:34653/ using document root <none> and password file <none>
I20260806 02:43:46.175611 20620 fs_manager.cc:359] Metadata directory not provided
I20260806 02:43:46.175690 20620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:43:46.179579 20620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:43:46.181636 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:46.182891 20620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:43:46.182984 20620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "f4cbf6a7d5666524be13be18ebb03542"
server_key_iv: "1d7b30c3b1c94bb180d705e0eef843bb"
server_key_version: "encryptionkey@0"
I20260806 02:43:46.183495 20620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:43:46.205250 20620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:43:46.208271 20620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:43:46.208431 20620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:43:46.208971 20620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:43:46.209800 20620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:43:46.209847 20620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:46.209888 20620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:43:46.209904 20620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:43:46.219022 20620 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:33951
I20260806 02:43:46.219053 20796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:33951 every 8 connection(s)
I20260806 02:43:46.220005 20620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:43:46.223309 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20620
Aug 06 02:43:46 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:43:46.232800 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.224032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.28.191.62:34267)
0806 02:43:46.224609 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.224612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.224623 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:43:46.224653 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.224655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.224687 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.224758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:46.225717 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.226276 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.227266 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.227624 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.227734 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.229355 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:46.229359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:46.229361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:46.229383 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:46.231041 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.231482 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.231486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.231488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.231576 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.231874 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.231877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.231879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.232078 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:46.232303 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.232462 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.232612 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":115,"threads_started":1}
I20260806 02:43:46.235366 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.221866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:37453 (local address 127.28.191.62:34267)
0806 02:43:46.222008 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.222014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.222780 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:43:46.224356 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.224360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.224421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.224513 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:46.226062 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.226734 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.228202 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.228582 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.228699 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.231548 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:46.231551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:46.231554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:46.231584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:46.233228 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.233744 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.233749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.233750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.233837 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.234197 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.234201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.234203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.234380 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:46.234497 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.235106 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.235221 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:43:46.236313 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.222180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:37453)
0806 02:43:46.222623 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.222667 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.224092 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.224716 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.224724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.225307 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:43:46.225920 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.225935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.226869 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.226872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.228068 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.228083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.228751 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.228754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.228898 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.229493 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:43:46.229509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:43:46.231382 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:43:46.233355 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:46.233361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:46.233376 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:46.233637 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:46.233942 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:43:46.233945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:43:46.233947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:43:46.234098 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:43:46.234640 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:43:46.234645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:43:46.234871 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.235063 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.235407 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260806 02:43:46.237423 20797 heartbeater.cc:347] Connected to a master server at 127.28.191.62:34267
I20260806 02:43:46.237681 20797 heartbeater.cc:464] Registering TS with master...
I20260806 02:43:46.238188 20797 heartbeater.cc:511] Master 127.28.191.62:34267 requested a full tablet report, sending...
I20260806 02:43:46.239240 20186 ts_manager.cc:194] Registered new tserver with Master: 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951)
I20260806 02:43:46.239845 20186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:37453
I20260806 02:43:46.246497 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:43:46.251029 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.247181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.28.191.62:34267)
0806 02:43:46.247342 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.247347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.247528 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:43:46.247732 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.247736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.247846 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.247963 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:46.248235 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.248759 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.249754 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.249992 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.250109 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.250452 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:46.250604 (+ 152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:43:46.250890 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41496: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:43:46.251151 20428 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41496: token expired
W20260806 02:43:46.251436 20086 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.191.62:34267)
I20260806 02:43:46.259205 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.251837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.28.191.62:34267)
0806 02:43:46.251959 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.251963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.252030 (+ 67us) server_negotiation.cc:407] Connection header received
0806 02:43:46.252215 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.252218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.252269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.252354 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:43:46.252991 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.253561 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.254380 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.254609 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.254717 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.256090 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:43:46.256094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:43:46.256096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:43:46.256120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:43:46.257501 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.257964 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.257968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.257970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.258056 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:43:46.258342 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:43:46.258346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:43:46.258347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:43:46.258526 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:43:46.258674 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.258926 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.259042 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:43:46.262094 20186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:43:46.265544 20186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:43:46.273974 20343 tablet_service.cc:1467] Processing CreateTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 (DEFAULT_TABLE table=test-table [id=a179cf00fd6f40c1bafa9bf999d6006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:46.274621 20343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f6c56a9b64445b3007e3557ebb6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:46.277634 20291 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap starting.
I20260806 02:43:46.278407 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.272744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:33951 (local address 127.0.0.1:35406)
0806 02:43:46.272913 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.272930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.273017 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.273964 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.273967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.273986 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:46.274106 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.274116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.275724 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.275727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.276822 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.276832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.277982 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.277986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.278088 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.278142 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.278200 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.278281 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:43:46.278582 20291 tablet_bootstrap.cc:654] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:46.279248 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.272972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.28.191.3:33951)
0806 02:43:46.273433 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.273448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.273474 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:43:46.273565 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.273571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.273797 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.273984 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:46.274270 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.275592 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.276951 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.277642 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.278723 (+ 1081us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.278903 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.278970 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.279088 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":170,"threads_started":1}
I20260806 02:43:46.280018 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.272991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.0.0.1:57128)
0806 02:43:46.273819 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.273834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.273930 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.274951 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.274954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.274973 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:46.275063 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.275069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.276456 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.276461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.277631 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.277643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.279257 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.279260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.279659 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.279691 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.279785 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.279875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":522,"threads_started":1}
I20260806 02:43:46.280088 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.273376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.28.191.2:40539)
0806 02:43:46.274322 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.274334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.274358 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:43:46.274432 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.274437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.274615 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.274776 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:46.275182 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.276320 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.278220 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.279125 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.279254 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.279406 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.279837 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.279934 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":164,"threads_started":1}
I20260806 02:43:46.280220 20291 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: No bootstrap required, opened a new log
I20260806 02:43:46.280304 20291 ts_tablet_manager.cc:1397] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:43:46.280766 20291 raft_consensus.cc:348] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.280897 20291 raft_consensus.cc:374] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:46.280948 20291 raft_consensus.cc:729] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebcb512d345431c9499f02e48a1048f, State: Initialized, Role: FOLLOWER
I20260806 02:43:46.282348 20708 tablet_service.cc:1467] Processing CreateTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 (DEFAULT_TABLE table=test-table [id=a179cf00fd6f40c1bafa9bf999d6006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:46.282943 20519 tablet_service.cc:1467] Processing CreateTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 (DEFAULT_TABLE table=test-table [id=a179cf00fd6f40c1bafa9bf999d6006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:43:46.283375 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f6c56a9b64445b3007e3557ebb6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:46.283805 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07f6c56a9b64445b3007e3557ebb6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:43:46.289330 20291 consensus_queue.cc:260] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.289692 20291 ts_tablet_manager.cc:1428] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20260806 02:43:46.291270 20817 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Bootstrap starting.
I20260806 02:43:46.291802 20816 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Bootstrap starting.
I20260806 02:43:46.294289 20817 tablet_bootstrap.cc:654] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:46.294382 20816 tablet_bootstrap.cc:654] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Neither blocks nor log segments found. Creating new log.
I20260806 02:43:46.295015 20817 log.cc:824] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:46.295100 20816 log.cc:824] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:43:46.297317 20817 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: No bootstrap required, opened a new log
I20260806 02:43:46.297538 20817 ts_tablet_manager.cc:1397] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:43:46.298497 20816 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: No bootstrap required, opened a new log
I20260806 02:43:46.298696 20816 ts_tablet_manager.cc:1397] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:43:46.301733 20817 raft_consensus.cc:348] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.302062 20817 raft_consensus.cc:374] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:46.302116 20817 raft_consensus.cc:729] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaae90fdedcb46ce9f3b295c682b9216, State: Initialized, Role: FOLLOWER
I20260806 02:43:46.302495 20816 raft_consensus.cc:348] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.302605 20817 consensus_queue.cc:260] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.302757 20816 raft_consensus.cc:374] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:43:46.302814 20816 raft_consensus.cc:729] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265b8a5f5f904a308e7e3d34b22dd9fd, State: Initialized, Role: FOLLOWER
I20260806 02:43:46.303277 20816 consensus_queue.cc:260] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.303669 20616 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
I20260806 02:43:46.303768 20817 ts_tablet_manager.cc:1428] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:43:46.304234 20816 ts_tablet_manager.cc:1428] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 02:43:46.304298 20797 heartbeater.cc:503] Master 127.28.191.62:34267 was elected leader, sending a full tablet report...
W20260806 02:43:46.321350 20426 tablet.cc:2369] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:43:46.322727 20432 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:43:46.322937 20432 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.324096 20432 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:46.329842 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.324463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:54445 (local address 127.28.191.2:40539)
0806 02:43:46.324734 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.324738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.324752 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:43:46.324935 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.324940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.325001 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.325145 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:46.325603 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.326473 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.328347 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.329030 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.329191 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.329250 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.329621 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.329706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260806 02:43:46.329842 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.324453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:40497 (local address 127.28.191.3:33951)
0806 02:43:46.324597 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.324602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.324883 (+ 281us) server_negotiation.cc:407] Connection header received
0806 02:43:46.324945 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.324949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.324997 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.325144 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:43:46.325602 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.326470 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.328347 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.329029 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.329191 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.329250 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.329621 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.329721 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:43:46.329852 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.324348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.28.191.1:54445)
0806 02:43:46.324618 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.324635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.324744 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.325166 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.325169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.325216 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:46.325460 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.325467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.326616 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.326620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.328197 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.328209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.329196 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.329200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.329411 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.329493 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.329584 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.329678 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":283,"spinlock_wait_cycles":39808}
I20260806 02:43:46.329861 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.324348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:33951 (local address 127.28.191.1:40497)
0806 02:43:46.324728 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.324742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:43:46.324833 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:43:46.325166 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:43:46.325169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:43:46.325215 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:43:46.325461 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.325467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.326616 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.326620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.328197 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:43:46.328209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.329201 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.329204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:43:46.329417 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.329493 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 02:43:46.329582 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:43:46.329677 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":344,"spinlock_wait_cycles":5504,"thread_start_us":116,"threads_started":1}
I20260806 02:43:46.330598 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f6c56a9b64445b3007e3557ebb6d7" candidate_uuid: "8ebcb512d345431c9499f02e48a1048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaae90fdedcb46ce9f3b295c682b9216" is_pre_election: true
I20260806 02:43:46.330745 20735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f6c56a9b64445b3007e3557ebb6d7" candidate_uuid: "8ebcb512d345431c9499f02e48a1048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" is_pre_election: true
I20260806 02:43:46.330950 20547 raft_consensus.cc:2454] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebcb512d345431c9499f02e48a1048f in term 0.
I20260806 02:43:46.331130 20735 raft_consensus.cc:2454] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebcb512d345431c9499f02e48a1048f in term 0.
I20260806 02:43:46.331540 20287 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f, eaae90fdedcb46ce9f3b295c682b9216; no voters:
I20260806 02:43:46.331898 20432 raft_consensus.cc:2781] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:43:46.331990 20432 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:43:46.332059 20432 raft_consensus.cc:3037] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:46.333293 20432 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.333716 20432 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 1 election: Requested vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
I20260806 02:43:46.334048 20735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f6c56a9b64445b3007e3557ebb6d7" candidate_uuid: "8ebcb512d345431c9499f02e48a1048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
I20260806 02:43:46.334123 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07f6c56a9b64445b3007e3557ebb6d7" candidate_uuid: "8ebcb512d345431c9499f02e48a1048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaae90fdedcb46ce9f3b295c682b9216"
I20260806 02:43:46.334172 20735 raft_consensus.cc:3037] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:46.334267 20547 raft_consensus.cc:3037] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:43:46.335500 20547 raft_consensus.cc:2454] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebcb512d345431c9499f02e48a1048f in term 1.
I20260806 02:43:46.335517 20735 raft_consensus.cc:2454] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebcb512d345431c9499f02e48a1048f in term 1.
I20260806 02:43:46.335938 20287 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f, eaae90fdedcb46ce9f3b295c682b9216; no voters:
I20260806 02:43:46.336146 20432 raft_consensus.cc:2781] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:43:46.336359 20432 raft_consensus.cc:686] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 LEADER]: Becoming Leader. State: Replica: 8ebcb512d345431c9499f02e48a1048f, State: Running, Role: LEADER
I20260806 02:43:46.336642 20432 consensus_queue.cc:237] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:43:46.339848 20186 catalog_manager.cc:5495] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ebcb512d345431c9499f02e48a1048f (127.28.191.1). New cstate: current_term: 1 leader_uuid: "8ebcb512d345431c9499f02e48a1048f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } health_report { overall_health: HEALTHY } } }
I20260806 02:43:46.360337 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.356749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.28.191.1:32895)
0806 02:43:46.356885 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:43:46.356889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:43:46.356901 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:43:46.356979 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:43:46.356981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:43:46.357038 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:43:46.357149 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:43:46.357570 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.358096 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.358961 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:43:46.359208 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:43:46.359296 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:43:46.359565 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:43:46.359981 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:43:46.360072 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:43:46.360171 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 02:43:46.360225 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:43:46.364382 20735 raft_consensus.cc:1261] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Refusing update from remote peer 8ebcb512d345431c9499f02e48a1048f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:46.364413 20547 raft_consensus.cc:1261] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Refusing update from remote peer 8ebcb512d345431c9499f02e48a1048f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:43:46.365231 20827 consensus_queue.cc:1035] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:46.365446 20432 consensus_queue.cc:1035] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [LEADER]: Connected to new peer: Peer: permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:43:46.372843 20832 mvcc.cc:205] Tried to move back new op lower bound from 7315391391182106624 to 7315391391078223872. Current Snapshot: MvccSnapshot[applied={T|T < 7315391391182106624}]
I20260806 02:43:46.377180 20835 mvcc.cc:205] Tried to move back new op lower bound from 7315391391182106624 to 7315391391078223872. Current Snapshot: MvccSnapshot[applied={T|T < 7315391391182106624 or (T in {7315391391182106624})}]
I20260806 02:43:46.381260 20185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:46.381440 20185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:43:46.384312 20185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Sending DeleteTablet for 3 replicas of tablet a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:43:46.385001 20708 tablet_service.cc:1514] Processing DeleteTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:46 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:35406
I20260806 02:43:46.385299 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:46 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:57128
I20260806 02:43:46.385551 20844 tablet_replica.cc:330] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: stopping tablet replica
I20260806 02:43:46.385730 20343 tablet_service.cc:1514] Processing DeleteTablet for tablet a07f6c56a9b64445b3007e3557ebb6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:43:46 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:44886
I20260806 02:43:46.385892 20844 raft_consensus.cc:2229] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:46.385906 20845 tablet_replica.cc:330] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: stopping tablet replica
I20260806 02:43:46.386225 20844 raft_consensus.cc:2258] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:46.386264 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 20236
I20260806 02:43:46.386286 20845 raft_consensus.cc:2229] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:43:46.386660 20845 raft_consensus.cc:2258] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:43:46.396768 20845 ts_tablet_manager.cc:1905] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:46.397401 20844 ts_tablet_manager.cc:1905] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:43:46.400077 20845 ts_tablet_manager.cc:1918] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:46.400208 20845 log.cc:1196] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:43:46.400188 20844 ts_tablet_manager.cc:1918] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:43:46.400285 20844 log.cc:1196] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:43:46.400609 20845 ts_tablet_manager.cc:1939] T a07f6c56a9b64445b3007e3557ebb6d7 P eaae90fdedcb46ce9f3b295c682b9216: Deleting consensus metadata
I20260806 02:43:46.400669 20844 ts_tablet_manager.cc:1939] T a07f6c56a9b64445b3007e3557ebb6d7 P 265b8a5f5f904a308e7e3d34b22dd9fd: Deleting consensus metadata
I20260806 02:43:46.402026 20170 catalog_manager.cc:4841] TS eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): tablet a07f6c56a9b64445b3007e3557ebb6d7 (table test-table [id=a179cf00fd6f40c1bafa9bf999d6006e]) successfully deleted
I20260806 02:43:46.402096 20168 catalog_manager.cc:4841] TS 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): tablet a07f6c56a9b64445b3007e3557ebb6d7 (table test-table [id=a179cf00fd6f40c1bafa9bf999d6006e]) successfully deleted
W20260806 02:43:46.402280 20167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:32895 (error 108)
I20260806 02:43:46.403246 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 20430
I20260806 02:43:46.403473 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:43:46.403131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:32895 (local address 127.0.0.1:34904)
0806 02:43:46.403281 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 02:43:46.403368 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:32895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:43:46.407339 20167 catalog_manager.cc:4584] TS 8ebcb512d345431c9499f02e48a1048f (127.28.191.1:32895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a07f6c56a9b64445b3007e3557ebb6d7: Network error: Client connection negotiation failed: client connection to 127.28.191.1:32895: connect: Connection refused (error 111)
I20260806 02:43:46.422533 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 20620
I20260806 02:43:46.433023 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 20137
I20260806 02:44:06.444496 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--webserver_interface=127.28.191.62
--webserver_port=42401
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:06.560830 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:06.561095 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:06.561134 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:06.565100 20848 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:44:06.565152 20848 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:44:06.565167 20848 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:44:06.565181 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:06.565199 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:06.565217 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:06.565277 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:06.565299 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:06.570462 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:06.571480 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:06.572476 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:06.581292 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:06.581298 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:06.581601 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:06.581614 20848 server_base.cc:1034] running on GCE node
I20260806 02:44:06.582269 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:06.583259 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:06.584455 20848 hybrid_clock.cc:648] HybridClock initialized: now 1785984246584429 us; error 45 us; skew 500 ppm
Aug 06 02:44:06 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984246, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:06.587371 20848 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:06.588194 20848 webserver.cc:460] Webserver started at http://127.28.191.62:42401/ using document root <none> and password file <none>
I20260806 02:44:06.588716 20848 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:06.588774 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:06.593169 20848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:44:06.595203 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:06.596418 20848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:06.596547 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "3f350a0295a44ffaaa7a171b9db8dfb6"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "b0b0303bb2415871ab75c64e9563d5dc"
server_key_iv: "a3ba1c5d55fdde4279ebc57abddfabd9"
server_key_version: "encryptionkey@0"
I20260806 02:44:06.597191 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:06.611594 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:06.615200 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:06.615406 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:06.623025 20848 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:34267
I20260806 02:44:06.623368 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:34267 every 8 connection(s)
I20260806 02:44:06.624150 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:06.628587 20932 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:44:06.630347 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap starting.
I20260806 02:44:06.632611 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20848
I20260806 02:44:06.633172 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:32895
--local_ip_for_outbound_sockets=127.28.191.1
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=37491
--webserver_interface=127.28.191.1
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:44:06.649547 20932 log.cc:824] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:06.661374 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:44:06.661814 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Bootstrap complete.
I20260806 02:44:06.666059 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:44:06.666443 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Initialized, Role: FOLLOWER
I20260806 02:44:06.666985 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:44:06.667127 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:06.667203 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:06.667311 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:44:06.669353 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:44:06.669687 20932 leader_election.cc:302] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f350a0295a44ffaaa7a171b9db8dfb6; no voters:
I20260806 02:44:06.670022 20932 leader_election.cc:288] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:44:06.670137 20937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:44:06.670354 20937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [term 3 LEADER]: Becoming Leader. State: Replica: 3f350a0295a44ffaaa7a171b9db8dfb6, State: Running, Role: LEADER
I20260806 02:44:06.670708 20937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } }
I20260806 02:44:06.671277 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:06.672159 20938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:44:06.672292 20938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:06.672693 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f350a0295a44ffaaa7a171b9db8dfb6. Latest consensus state: current_term: 3 leader_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f350a0295a44ffaaa7a171b9db8dfb6" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 34267 } } }
I20260806 02:44:06.672863 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:06.673303 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:06.678342 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:06.679047 20951 catalog_manager.cc:1259] Loaded cluster ID: bbe874944b6640cc86e658f1a1d7459d
I20260806 02:44:06.679109 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:06.681632 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:06.682392 20951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Loaded TSK: 0
I20260806 02:44:06.682479 20951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f350a0295a44ffaaa7a171b9db8dfb6: Loaded TSK: 1
F20260806 02:44:06.683261 20951 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9ccb5699ad google::LogMessage::Fail() at ??:0
@ 0x7f9ccb56c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9ccb5694de google::LogMessage::Flush() at ??:0
@ 0x7f9ccb56aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9cce22d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9cce225f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9cd5ece7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9cd5ea899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9cd5ea9178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9cd5eeadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9cd5f0982b std::function<>::operator()() at ??:0
@ 0x7f9cd5ea8b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9cd5ea9f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9cd5ea6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9cd5eea640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9ccd7bb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9ccd7bbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9ccd7bd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9ccd7ac512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9ccedf06db start_thread at ??:0
@ 0x7f9cca06761f clone at ??:0
W20260806 02:44:06.787804 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:06.788177 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:06.788264 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:06.794603 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:06.794687 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:06.794775 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:06.803011 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:32895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=37491
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:06.804479 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:06.805903 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:06.817835 20997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:06.817831 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:06.817817 21002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:06.818604 20935 server_base.cc:1034] running on GCE node
I20260806 02:44:06.819051 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:06.819757 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:06.820992 20935 hybrid_clock.cc:648] HybridClock initialized: now 1785984246820960 us; error 47 us; skew 500 ppm
Aug 06 02:44:06 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984246, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:06.824132 20935 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:44:06.825204 20935 webserver.cc:460] Webserver started at http://127.28.191.1:37491/ using document root <none> and password file <none>
I20260806 02:44:06.825814 20935 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:06.825914 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:06.830224 20935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:44:06.832785 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:06.834244 20935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:44:06.834406 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "8ebcb512d345431c9499f02e48a1048f"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "72a2f32712424f1e02e6856a3a757704"
server_key_iv: "dd75c72c27baacb1954638a7c4eac883"
server_key_version: "encryptionkey@0"
I20260806 02:44:06.834995 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:06.863981 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:06.867527 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:06.867750 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:06.868391 20935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:06.869620 21021 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:44:06.873944 20935 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:44:06.874018 20935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 02:44:06.874068 20935 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:44:06.875808 20935 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:44:06.875869 20935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:44:06.876127 21021 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap starting.
I20260806 02:44:06.887418 20935 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:32895
I20260806 02:44:06.887626 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:32895 every 8 connection(s)
I20260806 02:44:06.888718 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:06.892422 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 20935
I20260806 02:44:06.892977 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:40539
--local_ip_for_outbound_sockets=127.28.191.2
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=42933
--webserver_interface=127.28.191.2
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:44:06.899756 21021 log.cc:824] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:06.910090 21021 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:44:06.910562 21021 tablet_bootstrap.cc:492] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Bootstrap complete.
I20260806 02:44:06.912076 21021 ts_tablet_manager.cc:1397] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.012s
I20260806 02:44:06.915508 21021 raft_consensus.cc:348] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:06.916098 21021 raft_consensus.cc:729] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebcb512d345431c9499f02e48a1048f, State: Initialized, Role: FOLLOWER
I20260806 02:44:06.916646 21021 consensus_queue.cc:260] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:06.917794 21021 ts_tablet_manager.cc:1428] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260806 02:44:06.934027 21162 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:44:06.891093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:39699)
0806 02:44:06.891719 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:06.891786 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:06.895854 (+ 4068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:06.933267 (+ 37413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":181,"threads_started":1}
W20260806 02:44:06.934603 21159 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:34267 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:34267: Client connection negotiation failed: client connection to 127.28.191.62:34267: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:44:06.935411 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:06.935146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:38533)
0806 02:44:06.935288 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:06.935353 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:44:06.936072 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:06.935862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:39905)
0806 02:44:06.935973 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:06.936026 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:44:06.936244 21159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:44:07.023810 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:07.024134 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:07.024230 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:07.028290 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:07.028378 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:07.028465 21163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:07.034044 21163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:40539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=42933
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:07.035159 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:07.036288 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:07.046675 21171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:07.046664 21175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:07.046664 21173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:07.046984 21163 server_base.cc:1034] running on GCE node
I20260806 02:44:07.047482 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:07.048170 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:07.049336 21163 hybrid_clock.cc:648] HybridClock initialized: now 1785984247049299 us; error 49 us; skew 500 ppm
Aug 06 02:44:07 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984247, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:07.056327 21163 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:44:07.057391 21163 webserver.cc:460] Webserver started at http://127.28.191.2:42933/ using document root <none> and password file <none>
I20260806 02:44:07.057976 21163 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:07.058050 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:07.062091 21163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:07.064189 21185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.065573 21163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:44:07.065676 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "eaae90fdedcb46ce9f3b295c682b9216"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "574a801461588c3d2fbb4daace1e9229"
server_key_iv: "8f0ae35927ccd50d4e5508ad6f8be99a"
server_key_version: "encryptionkey@0"
I20260806 02:44:07.066208 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:07.084550 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:07.088065 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:07.088290 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:07.088912 21163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:07.089833 21163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:07.089906 21163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.089978 21163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:07.090029 21163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.098979 21163 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:40539
I20260806 02:44:07.099047 21339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:40539 every 8 connection(s)
I20260806 02:44:07.100016 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:07.102816 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 21163
I20260806 02:44:07.103299 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:33951
--local_ip_for_outbound_sockets=127.28.191.3
--tserver_master_addrs=127.28.191.62:34267
--webserver_port=34653
--webserver_interface=127.28.191.3
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:44:07.104075 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.102192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:43785)
0806 02:44:07.102708 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.103276 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
W20260806 02:44:07.104568 21340 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:34267 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:34267: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
I20260806 02:44:07.105443 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.105045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:40103)
0806 02:44:07.105234 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.105335 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:44:07.106160 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.105901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:41569)
0806 02:44:07.106027 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.106088 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:44:07.106349 21340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:44:07.217928 21345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:07.218178 21345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:07.218230 21345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:07.222194 21345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:07.222254 21345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:07.222340 21345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:07.227561 21345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:33951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=34653
--tserver_master_addrs=127.28.191.62:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:07.228600 21345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:07.229624 21345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:07.239153 21355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:07.239214 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:07.239153 21356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:07.239226 21345 server_base.cc:1034] running on GCE node
I20260806 02:44:07.239881 21345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:07.240592 21345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:07.241763 21345 hybrid_clock.cc:648] HybridClock initialized: now 1785984247241744 us; error 35 us; skew 500 ppm
Aug 06 02:44:07 dist-test-slave-tfpf krb5kdc[19367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984247, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:07.244727 21345 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:44:07.245698 21345 webserver.cc:460] Webserver started at http://127.28.191.3:34653/ using document root <none> and password file <none>
I20260806 02:44:07.246309 21345 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:07.246405 21345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:07.250258 21345 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 02:44:07.252406 21366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.253810 21345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:07.253968 21345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "265b8a5f5f904a308e7e3d34b22dd9fd"
format_stamp: "Formatted at 2026-08-06 02:43:24 on dist-test-slave-tfpf"
server_key: "f4cbf6a7d5666524be13be18ebb03542"
server_key_iv: "1d7b30c3b1c94bb180d705e0eef843bb"
server_key_version: "encryptionkey@0"
I20260806 02:44:07.254528 21345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:07.270677 21345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:07.274230 21345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:07.274452 21345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:07.275104 21345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:07.276000 21345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:07.276081 21345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.276155 21345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:07.276192 21345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:07.285521 21345 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:33951
I20260806 02:44:07.285552 21522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:33951 every 8 connection(s)
I20260806 02:44:07.286566 21345 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:07.290572 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.288890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:48423)
0806 02:44:07.289398 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.289829 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":146,"threads_started":1}
W20260806 02:44:07.291141 21523 heartbeater.cc:644] Failed to heartbeat to 127.28.191.62:34267 (0 consecutive failures): Network error: Failed to ping master at 127.28.191.62:34267: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
I20260806 02:44:07.291826 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.291555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:42991)
0806 02:44:07.291711 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.291762 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:44:07.291896 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 21345
I20260806 02:44:07.292608 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.292369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:50761)
0806 02:44:07.292492 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.292543 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:44:07.292838 21523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:44:07.937404 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:07.936877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:50047)
0806 02:44:07.937222 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:07.937283 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":79,"threads_started":1}
I20260806 02:44:08.107379 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.106976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:46777)
0806 02:44:08.107218 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:08.107276 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260806 02:44:08.293977 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.293533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:56363)
0806 02:44:08.293808 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:08.293869 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260806 02:44:08.675520 21533 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:08.675803 21533 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:08.677294 21533 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
Aug 06 02:44:08 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984246, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.2@KRBTEST.COM
Aug 06 02:44:08 dist-test-slave-tfpf krb5kdc[19367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984246, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.3@KRBTEST.COM
I20260806 02:44:08.692670 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.677755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:59945 (local address 127.28.191.2:40539)
0806 02:44:08.678334 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:08.678345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:08.678378 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:44:08.678991 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:08.678996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:08.679324 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:08.679637 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:08.680919 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.681973 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683273 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.683750 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683946 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:08.687161 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:08.687169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:08.687187 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:08.687228 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:08.689911 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:08.690503 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:08.690508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:08.690512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:08.690639 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:08.691016 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:08.691022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:08.691024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:08.691492 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:08.691658 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:08.692208 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 02:44:08.692475 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":146,"threads_started":1}
I20260806 02:44:08.692685 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.677580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40539 (local address 127.28.191.1:59945)
0806 02:44:08.677940 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:08.677957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:08.678082 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:08.679694 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:08.679701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:08.680247 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:08.680732 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:08.680749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.682163 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.682165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:08.683142 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:08.683153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.683896 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:08.684148 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:08.684898 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:08.684911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:08.686828 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:08.690089 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:08.690094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:08.690107 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:08.690390 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:08.690750 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:08.690754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:08.690756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:08.690914 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:08.691630 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:08.691637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:08.691875 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 02:44:08.692091 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:44:08.692457 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":170,"spinlock_wait_cycles":768,"thread_start_us":71,"threads_started":1}
I20260806 02:44:08.692736 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.677590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:33951 (local address 127.28.191.1:38803)
0806 02:44:08.677989 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:08.678004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:08.678088 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:08.679699 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:08.679703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:08.680246 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:08.680731 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:08.680750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.682161 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.682171 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:08.683141 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:08.683152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.683918 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:08.684182 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:08.684855 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:08.684873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:08.687039 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:08.690494 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:08.690496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:08.690497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:08.690665 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:08.691046 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:08.691049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:08.691051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:08.691137 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:08.691747 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:08.691750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:08.691878 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 02:44:08.692093 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 02:44:08.692541 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":336,"spinlock_wait_cycles":35072,"thread_start_us":60,"threads_started":1}
I20260806 02:44:08.693033 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.677760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:38803 (local address 127.28.191.3:33951)
0806 02:44:08.678203 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:08.678228 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:08.678270 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:44:08.678978 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:08.678985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:08.679325 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:08.679648 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:08.680934 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.681994 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683300 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:08.683778 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:08.683968 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:08.687275 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:08.687284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:08.687298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:08.687339 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:08.689916 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:08.690797 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:08.690805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:08.690815 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:08.690932 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:08.691240 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:08.691243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:08.691245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:08.691634 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:08.691777 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:08.692219 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 02:44:08.692514 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":102,"threads_started":1}
W20260806 02:44:08.695488 21014 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
W20260806 02:44:08.696012 21016 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:44:08.696126 21016 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f; no voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216
I20260806 02:44:08.696396 21533 raft_consensus.cc:2726] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:44:08.938637 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:08.938307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:53355)
0806 02:44:08.938465 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:08.938543 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:44:09.108932 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:09.108406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:44755)
0806 02:44:09.108723 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:09.108799 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":136,"threads_started":1}
I20260806 02:44:09.295471 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:09.294980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:34701)
0806 02:44:09.295299 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:09.295366 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260806 02:44:09.939977 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:09.939494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:57975)
0806 02:44:09.939821 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:09.939878 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
I20260806 02:44:10.110320 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:10.109870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:45879)
0806 02:44:10.110155 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:10.110219 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260806 02:44:10.296864 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:10.296385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:48127)
0806 02:44:10.296663 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:10.296741 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260806 02:44:10.892901 21547 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:10.893030 21547 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:10.893519 21547 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
W20260806 02:44:10.894469 21016 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
W20260806 02:44:10.894598 21014 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:44:10.894660 21014 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f; no voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216
I20260806 02:44:10.894809 21547 raft_consensus.cc:2726] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:44:10.941352 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:10.940769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:50793)
0806 02:44:10.941165 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:10.941231 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260806 02:44:11.111780 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:11.111252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:33803)
0806 02:44:11.111604 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:11.111676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260806 02:44:11.298595 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:11.298089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:34237)
0806 02:44:11.298419 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:11.298487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260806 02:44:11.942781 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:11.942253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:43201)
0806 02:44:11.942610 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:11.942675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
I20260806 02:44:12.113399 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:12.112814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:42229)
0806 02:44:12.113211 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:12.113287 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":87,"threads_started":1}
I20260806 02:44:12.300069 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:12.299572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:57625)
0806 02:44:12.299896 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:12.299964 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260806 02:44:12.944195 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:12.943662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:36803)
0806 02:44:12.944012 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:12.944082 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
I20260806 02:44:13.114928 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:13.114446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:43629)
0806 02:44:13.114756 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:13.114825 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260806 02:44:13.292678 21562 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:13.292815 21562 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:13.293320 21562 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
W20260806 02:44:13.294142 21014 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
W20260806 02:44:13.294257 21016 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:44:13.294353 21014 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f; no voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216
I20260806 02:44:13.294515 21562 raft_consensus.cc:2726] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:44:13.301476 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:13.300955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:36469)
0806 02:44:13.301314 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:13.301377 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":186,"threads_started":1}
I20260806 02:44:13.945675 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:13.945217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:58763)
0806 02:44:13.945510 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:13.945575 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260806 02:44:14.116484 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:14.115905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:32851)
0806 02:44:14.116275 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:14.116362 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
I20260806 02:44:14.302963 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:14.302411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:44975)
0806 02:44:14.302793 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:14.302862 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260806 02:44:14.947099 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:14.946600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:45577)
0806 02:44:14.946922 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:14.946989 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260806 02:44:15.117770 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:15.117363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:42381)
0806 02:44:15.117614 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:15.117669 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260806 02:44:15.304363 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:15.303907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:51543)
0806 02:44:15.304191 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:15.304258 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260806 02:44:15.948494 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:15.948002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:51563)
0806 02:44:15.948315 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:15.948379 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260806 02:44:16.119146 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:16.118650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:50875)
0806 02:44:16.118928 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:16.119040 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260806 02:44:16.305881 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:16.305362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:36573)
0806 02:44:16.305697 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:16.305765 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260806 02:44:16.950071 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:16.949582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:42495)
0806 02:44:16.949887 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:16.949956 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260806 02:44:17.120836 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:17.120235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:60057)
0806 02:44:17.120628 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:17.120687 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":114,"threads_started":1}
I20260806 02:44:17.307250 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:17.306750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:58447)
0806 02:44:17.307065 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:17.307131 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260806 02:44:17.813253 21581 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:17.813398 21581 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:17.813882 21581 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
W20260806 02:44:17.814813 21016 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
W20260806 02:44:17.814956 21014 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:44:17.815094 21014 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f; no voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216
I20260806 02:44:17.815300 21581 raft_consensus.cc:2726] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:44:17.951540 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:17.951038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:46029)
0806 02:44:17.951355 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:17.951427 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260806 02:44:18.122036 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:18.121608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:50739)
0806 02:44:18.121879 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:18.121938 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260806 02:44:18.308475 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:18.308054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:47023)
0806 02:44:18.308319 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:18.308377 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260806 02:44:18.952798 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:18.952378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:44963)
0806 02:44:18.952640 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:18.952697 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260806 02:44:19.123301 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:19.122860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:53539)
0806 02:44:19.123142 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:19.123200 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":134,"threads_started":1}
I20260806 02:44:19.309873 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:19.309440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:42229)
0806 02:44:19.309714 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:19.309773 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260806 02:44:19.954104 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:19.953684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:35763)
0806 02:44:19.953949 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:19.954004 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260806 02:44:20.124663 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:20.124221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:33327)
0806 02:44:20.124508 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:20.124566 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260806 02:44:20.311133 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:20.310686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:50699)
0806 02:44:20.310976 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:20.311036 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260806 02:44:20.853500 21597 raft_consensus.cc:482] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:20.853684 21597 raft_consensus.cc:504] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "265b8a5f5f904a308e7e3d34b22dd9fd" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 33951 } } peers { permanent_uuid: "eaae90fdedcb46ce9f3b295c682b9216" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40539 } } peers { permanent_uuid: "8ebcb512d345431c9499f02e48a1048f" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 32895 } }
I20260806 02:44:20.854180 21597 leader_election.cc:288] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951), eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539)
W20260806 02:44:20.854929 21016 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaae90fdedcb46ce9f3b295c682b9216 (127.28.191.2:40539): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
W20260806 02:44:20.855041 21014 leader_election.cc:341] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 265b8a5f5f904a308e7e3d34b22dd9fd (127.28.191.3:33951): Not found: Tablet not found: a07f6c56a9b64445b3007e3557ebb6d7
I20260806 02:44:20.855175 21014 leader_election.cc:302] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ebcb512d345431c9499f02e48a1048f; no voters: 265b8a5f5f904a308e7e3d34b22dd9fd, eaae90fdedcb46ce9f3b295c682b9216
I20260806 02:44:20.855370 21597 raft_consensus.cc:2726] T a07f6c56a9b64445b3007e3557ebb6d7 P 8ebcb512d345431c9499f02e48a1048f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:44:20.955498 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:20.954970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:33815)
0806 02:44:20.955324 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:20.955401 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":117,"threads_started":1}
I20260806 02:44:21.125844 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:21.125417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:50333)
0806 02:44:21.125696 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:21.125748 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260806 02:44:21.312333 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:21.311901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:40083)
0806 02:44:21.312179 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:21.312234 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260806 02:44:21.956992 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:21.956569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.1:51389)
0806 02:44:21.956840 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:21.956895 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260806 02:44:22.127220 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:22.126758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.2:57387)
0806 02:44:22.127021 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:22.127114 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260806 02:44:22.300740 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 20935
I20260806 02:44:22.319175 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:22.315112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:34267 (local address 127.28.191.3:35289)
0806 02:44:22.318896 (+ 3784us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:22.318990 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.62:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3649,"thread_start_us":716,"threads_started":1}
I20260806 02:44:22.325570 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 21163
I20260806 02:44:22.338568 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 21345
2026-08-06T02:44:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59731 ms)
[----------] 30 tests from SecurityITest (246100 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:22 dist-test-slave-tfpf krb5kdc[21608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:22 dist-test-slave-tfpf krb5kdc[21608](info): set up 2 sockets
Aug 06 02:44:22 dist-test-slave-tfpf krb5kdc[21608](info): commencing operation
krb5kdc: starting...
W20260806 02:44:23.664462 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:23 dist-test-slave-tfpf krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:23.796063 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38351
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:38351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:23.909837 21623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:23.910096 21623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:23.910136 21623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:23.914085 21623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:23.914135 21623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:23.914155 21623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:23.914170 21623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:23.914228 21623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:23.919445 21623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:38351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:38351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:23.920476 21623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:23.921500 21623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:23.930244 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:23.930353 21623 server_base.cc:1034] running on GCE node
W20260806 02:44:23.930275 21629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:23.930274 21628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:23.931094 21623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:23.932130 21623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:23.933285 21623 hybrid_clock.cc:648] HybridClock initialized: now 1785984263933258 us; error 40 us; skew 500 ppm
Aug 06 02:44:23 dist-test-slave-tfpf krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984263, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:23.935982 21623 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:23.936880 21623 webserver.cc:460] Webserver started at http://127.28.191.62:44589/ using document root <none> and password file <none>
I20260806 02:44:23.937469 21623 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:23.937530 21623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:23.937705 21623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:23.939666 21623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "819e1102d4f34dbe8c488d3e283b1b88"
format_stamp: "Formatted at 2026-08-06 02:44:23 on dist-test-slave-tfpf"
server_key: "5740a912b6a6451b098d004a49b060cf"
server_key_iv: "d8ae8d98f95b0a070d19bf5c31fee43d"
server_key_version: "encryptionkey@0"
I20260806 02:44:23.940191 21623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "819e1102d4f34dbe8c488d3e283b1b88"
format_stamp: "Formatted at 2026-08-06 02:44:23 on dist-test-slave-tfpf"
server_key: "5740a912b6a6451b098d004a49b060cf"
server_key_iv: "d8ae8d98f95b0a070d19bf5c31fee43d"
server_key_version: "encryptionkey@0"
I20260806 02:44:23.943820 21623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:44:23.946259 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:23.947490 21623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:44:23.947631 21623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "819e1102d4f34dbe8c488d3e283b1b88"
format_stamp: "Formatted at 2026-08-06 02:44:23 on dist-test-slave-tfpf"
server_key: "5740a912b6a6451b098d004a49b060cf"
server_key_iv: "d8ae8d98f95b0a070d19bf5c31fee43d"
server_key_version: "encryptionkey@0"
I20260806 02:44:23.947737 21623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:23.965312 21623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:23.968334 21623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:23.968554 21623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:23.976053 21623 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:38351
I20260806 02:44:23.976083 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:38351 every 8 connection(s)
I20260806 02:44:23.977052 21623 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:23.980051 21709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:23.982810 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 21623
I20260806 02:44:23.982961 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:23.983316 29436 external_mini_cluster.cc:1428] Setting key 7d6a83389c8c6f3123a72a60639a4ae5
I20260806 02:44:23.986375 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: Bootstrap starting.
I20260806 02:44:23.989027 21709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:23.989998 21709 log.cc:824] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:23.992524 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: No bootstrap required, opened a new log
Aug 06 02:44:23 dist-test-slave-tfpf krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:23.996089 21709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } }
I20260806 02:44:23.996397 21709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:23.996484 21709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819e1102d4f34dbe8c488d3e283b1b88, State: Initialized, Role: FOLLOWER
I20260806 02:44:23.997047 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } }
I20260806 02:44:23.997283 21709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:23.997377 21709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:23.997481 21709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:23.998733 21709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } }
I20260806 02:44:23.999107 21709 leader_election.cc:302] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819e1102d4f34dbe8c488d3e283b1b88; no voters:
I20260806 02:44:23.999452 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:23.984760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.28.191.62:38351)
0806 02:44:23.985232 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:23.985246 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:23.985280 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:44:23.985906 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:23.985922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:23.986284 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:23.986618 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:23.987545 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:23.988568 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:23.989858 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:23.990442 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:23.990629 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:23.993570 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:23.993575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:23.993590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:23.993619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:23.996069 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:23.996608 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:23.996621 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:23.996627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:23.996790 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:23.997163 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:23.997166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:23.997167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:23.997732 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:23.997894 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:23.998203 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:44:23.998532 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":129,"threads_started":1}
I20260806 02:44:23.999469 21709 leader_election.cc:288] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:23.999570 21714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:23.999768 21714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [term 1 LEADER]: Becoming Leader. State: Replica: 819e1102d4f34dbe8c488d3e283b1b88, State: Running, Role: LEADER
I20260806 02:44:24.000157 21714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } }
I20260806 02:44:24.000852 21709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:24.002995 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819e1102d4f34dbe8c488d3e283b1b88. Latest consensus state: current_term: 1 leader_uuid: "819e1102d4f34dbe8c488d3e283b1b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } } }
I20260806 02:44:24.003153 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:24.003396 21716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819e1102d4f34dbe8c488d3e283b1b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819e1102d4f34dbe8c488d3e283b1b88" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 38351 } } }
I20260806 02:44:24.003504 21716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:24.003582 21731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:24.006443 21731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:24.012120 21731 catalog_manager.cc:1347] Generated new cluster ID: e96f23e16e344d82b96ff4ceec0a2d63
I20260806 02:44:24.012197 21731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:24.025862 21731 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:24.026793 21731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:24.034112 21731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: Generated new TSK 0
I20260806 02:44:24.034829 21731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:44:24.076205 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:24.190922 21739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:24.191206 21739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:24.191282 21739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:24.195248 21739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:24.195329 21739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:24.195423 21739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:24.200639 21739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:24.201783 21739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:24.202936 21739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:24.212455 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.212473 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.212462 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:24.212710 21739 server_base.cc:1034] running on GCE node
I20260806 02:44:24.213220 21739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:24.213874 21739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:24.215075 21739 hybrid_clock.cc:648] HybridClock initialized: now 1785984264215042 us; error 43 us; skew 500 ppm
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:24.217983 21739 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:44:24.218778 21739 webserver.cc:460] Webserver started at http://127.28.191.1:36893/ using document root <none> and password file <none>
I20260806 02:44:24.219305 21739 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:24.219365 21739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:24.219527 21739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:24.221527 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "a95269497ed247d79b092f11380f675e"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "bf4a42d734dcc979ef3227aa0938e710"
server_key_iv: "b6fd3a7f4a6fbf59ea27559887374af6"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.222051 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "a95269497ed247d79b092f11380f675e"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "bf4a42d734dcc979ef3227aa0938e710"
server_key_iv: "b6fd3a7f4a6fbf59ea27559887374af6"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.225739 21739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:44:24.228003 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.229247 21739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:24.229341 21739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "a95269497ed247d79b092f11380f675e"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "bf4a42d734dcc979ef3227aa0938e710"
server_key_iv: "b6fd3a7f4a6fbf59ea27559887374af6"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.229434 21739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:24.243513 21739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:24.246531 21739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:24.246703 21739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:24.247234 21739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:24.248025 21739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:24.248075 21739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.248114 21739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:24.248131 21739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.257738 21739 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:35435
I20260806 02:44:24.257797 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:35435 every 8 connection(s)
I20260806 02:44:24.258680 21739 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:24.263499 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 21739
I20260806 02:44:24.263648 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:24.263980 29436 external_mini_cluster.cc:1428] Setting key 956068fd1ef6e353c5180d802312cd3a
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:24.274134 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.260525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:51813 (local address 127.28.191.62:38351)
0806 02:44:24.260699 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.260704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.261806 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 02:44:24.262656 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.262659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.262708 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.262787 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:24.264360 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.265036 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.266509 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.266749 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.266875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.269980 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:24.269984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:24.269986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:24.270010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:24.271674 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.272247 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.272253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.272255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.272353 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.272751 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.272755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.272757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.272960 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:24.273112 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.273788 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.273924 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:44:24.274981 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.260933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38351 (local address 127.28.191.1:51813)
0806 02:44:24.261666 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.261707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.262454 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.262958 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.262965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.263468 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:24.264174 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.264196 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.265246 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.265251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.266361 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.266373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.266918 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.266923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.267109 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.267854 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:24.267879 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:24.269785 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:24.271820 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.271826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.271841 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.272131 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.272466 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.272469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.272479 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.272633 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.273134 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:24.273142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:24.273506 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.273740 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.274073 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":192,"threads_started":1}
I20260806 02:44:24.276315 21903 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38351
I20260806 02:44:24.276584 21903 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:44:24.277205 21903 heartbeater.cc:511] Master 127.28.191.62:38351 requested a full tablet report, sending...
I20260806 02:44:24.278857 21664 ts_manager.cc:194] Registered new tserver with Master: a95269497ed247d79b092f11380f675e (127.28.191.1:35435)
I20260806 02:44:24.279974 21664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:51813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:44:24.293213 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:24.410197 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:24.410435 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:24.410475 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:24.414435 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:24.414490 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:24.414546 21908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:24.419727 21908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:24.420723 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:24.421728 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:24.431228 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.431232 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.431232 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:24.431303 21908 server_base.cc:1034] running on GCE node
I20260806 02:44:24.431879 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:24.432511 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:24.433718 21908 hybrid_clock.cc:648] HybridClock initialized: now 1785984264433688 us; error 44 us; skew 500 ppm
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:24.436443 21908 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:44:24.437399 21908 webserver.cc:460] Webserver started at http://127.28.191.2:34561/ using document root <none> and password file <none>
I20260806 02:44:24.437955 21908 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:24.438069 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:24.438333 21908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:24.440390 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "4b007b4a6b7c4ea1a962f954434f8d97"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "172dceafb11d5dbb9c2bcccc5149f6fa"
server_key_iv: "d7f3dd34d2a57478e699448662a3ec4d"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.440981 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "4b007b4a6b7c4ea1a962f954434f8d97"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "172dceafb11d5dbb9c2bcccc5149f6fa"
server_key_iv: "d7f3dd34d2a57478e699448662a3ec4d"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.444530 21908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:44:24.446907 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.448077 21908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:24.448225 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "4b007b4a6b7c4ea1a962f954434f8d97"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "172dceafb11d5dbb9c2bcccc5149f6fa"
server_key_iv: "d7f3dd34d2a57478e699448662a3ec4d"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.448339 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:24.459447 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:24.462309 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:24.462512 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:24.463088 21908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:24.463902 21908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:24.463977 21908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.464051 21908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:24.464097 21908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.473531 21908 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:35217
I20260806 02:44:24.473563 22078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:35217 every 8 connection(s)
I20260806 02:44:24.474503 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:24.481223 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 21908
I20260806 02:44:24.481387 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:44:24.481709 29436 external_mini_cluster.cc:1428] Setting key 3d07e4859b377791b601e6e67b63dcd0
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:24.487721 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.476251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:40787 (local address 127.28.191.62:38351)
0806 02:44:24.476393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.476396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.477413 (+ 1017us) server_negotiation.cc:407] Connection header received
0806 02:44:24.478280 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.478283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.478357 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.478456 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:24.479543 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.480066 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.481266 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.481598 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.481708 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.484217 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:24.484221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:24.484224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:24.484252 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:24.485726 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.486219 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.486224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.486225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.486309 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.486651 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.486655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.486657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.486843 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:24.486953 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.487435 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.487555 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:44:24.488621 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.476588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38351 (local address 127.28.191.2:40787)
0806 02:44:24.477266 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.477306 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.478074 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.478631 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.478638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.478964 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:24.479392 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.479405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.480211 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.480221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.481146 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.481156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.481755 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.481759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.481935 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.482587 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:24.482603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:24.484064 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:24.485844 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.485850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.485863 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.486115 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.486403 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.486408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.486410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.486547 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.486953 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:24.486961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:24.487199 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.487379 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.487691 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":111,"threads_started":1}
I20260806 02:44:24.489907 22079 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38351
I20260806 02:44:24.490176 22079 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:24.490785 22079 heartbeater.cc:511] Master 127.28.191.62:38351 requested a full tablet report, sending...
I20260806 02:44:24.491998 21664 ts_manager.cc:194] Registered new tserver with Master: 4b007b4a6b7c4ea1a962f954434f8d97 (127.28.191.2:35217)
I20260806 02:44:24.492574 21664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:40787
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:44:24.511938 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:24.626819 22084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:24.627110 22084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:24.627218 22084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:24.631264 22084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:24.631374 22084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:24.631459 22084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:24.636772 22084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:24.637811 22084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:24.638780 22084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:24.648085 22092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.648085 22095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:24.648099 22091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:24.648561 22084 server_base.cc:1034] running on GCE node
I20260806 02:44:24.649019 22084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:24.649832 22084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:24.651023 22084 hybrid_clock.cc:648] HybridClock initialized: now 1785984264651017 us; error 37 us; skew 500 ppm
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:24.653997 22084 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:44:24.654827 22084 webserver.cc:460] Webserver started at http://127.28.191.3:41301/ using document root <none> and password file <none>
I20260806 02:44:24.655367 22084 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:24.655458 22084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:24.655695 22084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:24.657757 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "fcf236ef707f4bd7adc496f72f05afc7"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "e5e14a42a97fc5bdebaf1d3af6ee3e4b"
server_key_iv: "132af23913e9a1ec5161cddb26034d76"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.658362 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "fcf236ef707f4bd7adc496f72f05afc7"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "e5e14a42a97fc5bdebaf1d3af6ee3e4b"
server_key_iv: "132af23913e9a1ec5161cddb26034d76"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.662025 22084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:24.664534 22105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.665832 22084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:24.666005 22084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "fcf236ef707f4bd7adc496f72f05afc7"
format_stamp: "Formatted at 2026-08-06 02:44:24 on dist-test-slave-tfpf"
server_key: "e5e14a42a97fc5bdebaf1d3af6ee3e4b"
server_key_iv: "132af23913e9a1ec5161cddb26034d76"
server_key_version: "encryptionkey@0"
I20260806 02:44:24.666127 22084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:24.684695 22084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:24.688544 22084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:24.688784 22084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:24.689543 22084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:24.690465 22084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:24.690557 22084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.690632 22084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:24.690690 22084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:24.700372 22254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:46245 every 8 connection(s)
I20260806 02:44:24.700380 22084 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:46245
I20260806 02:44:24.701630 22084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:24.709580 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 22084
I20260806 02:44:24.709750 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:24.710067 29436 external_mini_cluster.cc:1428] Setting key cfcb60688355ef97c1853710dcc41461
Aug 06 02:44:24 dist-test-slave-tfpf krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984264, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:24.715135 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.703584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:42391 (local address 127.28.191.62:38351)
0806 02:44:24.703726 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.703730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.704660 (+ 930us) server_negotiation.cc:407] Connection header received
0806 02:44:24.705554 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.705558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.705608 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.705684 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:24.706801 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.707333 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.708408 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.708621 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.708700 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.711673 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:24.711678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:24.711682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:24.711716 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:24.713371 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.713832 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.713834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.713835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.713907 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.714194 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.714196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.714198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.714380 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:24.714457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.714899 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.714991 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:24.715901 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.703845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:38351 (local address 127.28.191.3:42391)
0806 02:44:24.704507 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.704548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.705365 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.705845 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.705852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.706212 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:24.706672 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.706685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.707439 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.707441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.708295 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.708303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.708750 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.708752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.708915 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.709620 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:24.709635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:24.711485 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:24.713473 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.713478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.713488 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.713745 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.713985 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:24.713987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:24.713988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:24.714114 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:24.714469 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:24.714474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:24.714684 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.714849 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.715078 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":109,"threads_started":1}
I20260806 02:44:24.716993 22255 heartbeater.cc:347] Connected to a master server at 127.28.191.62:38351
I20260806 02:44:24.717253 22255 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:24.717713 22255 heartbeater.cc:511] Master 127.28.191.62:38351 requested a full tablet report, sending...
I20260806 02:44:24.718714 21664 ts_manager.cc:194] Registered new tserver with Master: fcf236ef707f4bd7adc496f72f05afc7 (127.28.191.3:46245)
I20260806 02:44:24.719233 21664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:42391
I20260806 02:44:24.723906 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:24.733342 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.725922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.28.191.62:38351)
0806 02:44:24.726047 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.726052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.726276 (+ 224us) server_negotiation.cc:407] Connection header received
0806 02:44:24.726425 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.726428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.726471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.726571 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:24.727262 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.727766 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.728621 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.728845 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.728954 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.730316 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:24.730320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:24.730322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:24.730346 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:24.731734 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.732154 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.732158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.732160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.732243 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:24.732520 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:24.732524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:24.732526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:24.732734 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:24.732882 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.733147 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.733239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:44:24.736163 21664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:24.739180 21664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:24.754031 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.747568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.28.191.1:35435)
0806 02:44:24.748195 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.748209 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.748602 (+ 393us) server_negotiation.cc:407] Connection header received
0806 02:44:24.748835 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.748849 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.748998 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.749178 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:24.749883 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.751000 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.752530 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.753401 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.753511 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.753657 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.753795 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.753890 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":132,"threads_started":1}
I20260806 02:44:24.754024 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.747305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:35435 (local address 127.0.0.1:41904)
0806 02:44:24.748495 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.748508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.748679 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.749377 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.749380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.749394 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:24.749757 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.749766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.751155 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.751159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.752426 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.752436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.753514 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.753517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.753601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.753631 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.753755 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.753874 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":102,"spinlock_wait_cycles":1280,"thread_start_us":90,"threads_started":1}
I20260806 02:44:24.755252 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.747267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:46245 (local address 127.0.0.1:55392)
0806 02:44:24.748094 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.748141 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.748261 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.749234 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.749240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.749262 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:24.749392 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.749398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.750902 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.750906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.752425 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.752436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.754855 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.754858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.754949 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.754972 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.755041 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.755114 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":62,"thread_start_us":73,"threads_started":1}
I20260806 02:44:24.755496 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.748107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:35217 (local address 127.0.0.1:60722)
0806 02:44:24.748553 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:24.748570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:24.748668 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:24.749355 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:24.749360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:24.749382 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:24.749761 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.749766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.752453 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.752458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.753649 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:24.753660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.755104 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.755108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:24.755197 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.755228 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:44:24.755305 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:44:24.755371 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":220,"thread_start_us":87,"threads_started":1}
I20260806 02:44:24.755602 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.748473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.28.191.2:35217)
0806 02:44:24.748907 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.748925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.748944 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:44:24.749011 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.749016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.749207 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.749520 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:24.751189 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.752303 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.753788 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.754957 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.755081 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.755206 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.755357 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.755460 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":121,"threads_started":1}
I20260806 02:44:24.755728 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:24.747372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.28.191.3:46245)
0806 02:44:24.748166 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:24.748178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:24.748206 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:44:24.748833 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:24.748840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:24.749033 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:24.749224 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:24.749513 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.750692 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.753722 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:24.754514 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:24.754640 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:24.754787 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:24.755477 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 02:44:24.755577 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":168,"threads_started":1}
I20260806 02:44:24.757454 21819 tablet_service.cc:1467] Processing CreateTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca (DEFAULT_TABLE table=test-table [id=a64f2624e5f647b586ddff21b69aac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:24.758528 21819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f050308d4a4cf0a6dd2f472eb21cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:24.758255 22167 tablet_service.cc:1467] Processing CreateTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca (DEFAULT_TABLE table=test-table [id=a64f2624e5f647b586ddff21b69aac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:24.758256 21991 tablet_service.cc:1467] Processing CreateTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca (DEFAULT_TABLE table=test-table [id=a64f2624e5f647b586ddff21b69aac96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:24.759234 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f050308d4a4cf0a6dd2f472eb21cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:24.759250 22167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f050308d4a4cf0a6dd2f472eb21cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:24.764750 22283 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Bootstrap starting.
I20260806 02:44:24.767050 22286 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Bootstrap starting.
I20260806 02:44:24.767143 22283 tablet_bootstrap.cc:654] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:24.768222 22283 log.cc:824] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:24.768846 22284 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Bootstrap starting.
I20260806 02:44:24.768885 22286 tablet_bootstrap.cc:654] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:24.770059 22286 log.cc:824] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:24.770682 22284 tablet_bootstrap.cc:654] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:24.771767 22284 log.cc:824] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:24.771936 22286 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: No bootstrap required, opened a new log
I20260806 02:44:24.772130 22286 ts_tablet_manager.cc:1397] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:44:24.773391 22283 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: No bootstrap required, opened a new log
I20260806 02:44:24.773602 22283 ts_tablet_manager.cc:1397] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 02:44:24.773993 22284 tablet_bootstrap.cc:492] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: No bootstrap required, opened a new log
I20260806 02:44:24.774191 22284 ts_tablet_manager.cc:1397] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:44:24.776105 22286 raft_consensus.cc:348] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.776389 22286 raft_consensus.cc:374] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:24.776468 22286 raft_consensus.cc:729] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf236ef707f4bd7adc496f72f05afc7, State: Initialized, Role: FOLLOWER
I20260806 02:44:24.776835 22283 raft_consensus.cc:348] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.776963 22286 consensus_queue.cc:260] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.777050 22283 raft_consensus.cc:374] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:24.777129 22283 raft_consensus.cc:729] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95269497ed247d79b092f11380f675e, State: Initialized, Role: FOLLOWER
I20260806 02:44:24.777596 22283 consensus_queue.cc:260] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.777715 22255 heartbeater.cc:503] Master 127.28.191.62:38351 was elected leader, sending a full tablet report...
I20260806 02:44:24.778040 22286 ts_tablet_manager.cc:1428] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:44:24.777988 22284 raft_consensus.cc:348] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.778250 22284 raft_consensus.cc:374] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:24.778314 22284 raft_consensus.cc:729] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b007b4a6b7c4ea1a962f954434f8d97, State: Initialized, Role: FOLLOWER
I20260806 02:44:24.778462 22283 ts_tablet_manager.cc:1428] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:44:24.778821 21903 heartbeater.cc:503] Master 127.28.191.62:38351 was elected leader, sending a full tablet report...
I20260806 02:44:24.779058 22284 consensus_queue.cc:260] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:24.779980 22284 ts_tablet_manager.cc:1428] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:44:24.780306 22079 heartbeater.cc:503] Master 127.28.191.62:38351 was elected leader, sending a full tablet report...
W20260806 02:44:24.952610 22256 tablet.cc:2369] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:44:24.975613 22080 tablet.cc:2369] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:44:25.010012 21904 tablet.cc:2369] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:25.104089 22294 raft_consensus.cc:482] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:25.104343 22294 raft_consensus.cc:504] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:25.105664 22294 leader_election.cc:288] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a95269497ed247d79b092f11380f675e (127.28.191.1:35435), 4b007b4a6b7c4ea1a962f954434f8d97 (127.28.191.2:35217)
I20260806 02:44:25.110327 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.105952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:35217 (local address 127.28.191.3:59401)
0806 02:44:25.106094 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:25.106110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:25.106216 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:25.106520 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:25.106523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:25.106551 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:25.106678 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.106685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.107848 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.107851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.109001 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.109011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.109859 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.109863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.109971 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.109998 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:44:25.110081 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:44:25.110164 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 02:44:25.110343 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.106084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:59401 (local address 127.28.191.2:35217)
0806 02:44:25.106238 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.106244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.106263 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:44:25.106323 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.106326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.106380 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.106468 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:25.106908 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.107720 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.109174 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.109690 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.109867 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.109927 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.110114 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.110187 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:44:25.110757 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.106092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60697 (local address 127.28.191.1:35435)
0806 02:44:25.106214 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.106218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.106860 (+ 642us) server_negotiation.cc:407] Connection header received
0806 02:44:25.106928 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.106931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.107022 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.107136 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:25.107459 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.108110 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.109451 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.110064 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.110170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.110225 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.110556 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.110644 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:44:25.111236 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.105952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:35435 (local address 127.28.191.3:60697)
0806 02:44:25.106692 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:25.106705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:25.106810 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:25.107174 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:25.107177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:25.107197 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:25.107308 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.107312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.108265 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.108268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.109331 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.109339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.110278 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.110281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.110402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.110425 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:44:25.110944 (+ 519us) client_negotiation.cc:240] Negotiation successful
0806 02:44:25.111031 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":335,"thread_start_us":152,"threads_started":1}
I20260806 02:44:25.111164 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f050308d4a4cf0a6dd2f472eb21cca" candidate_uuid: "fcf236ef707f4bd7adc496f72f05afc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" is_pre_election: true
I20260806 02:44:25.111513 22018 raft_consensus.cc:2454] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcf236ef707f4bd7adc496f72f05afc7 in term 0.
I20260806 02:44:25.111905 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f050308d4a4cf0a6dd2f472eb21cca" candidate_uuid: "fcf236ef707f4bd7adc496f72f05afc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95269497ed247d79b092f11380f675e" is_pre_election: true
I20260806 02:44:25.112036 22110 leader_election.cc:302] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b007b4a6b7c4ea1a962f954434f8d97, fcf236ef707f4bd7adc496f72f05afc7; no voters:
I20260806 02:44:25.112248 21840 raft_consensus.cc:2454] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcf236ef707f4bd7adc496f72f05afc7 in term 0.
I20260806 02:44:25.112419 22294 raft_consensus.cc:2781] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:25.112517 22294 raft_consensus.cc:482] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:44:25.112596 22294 raft_consensus.cc:3037] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:25.113722 22294 raft_consensus.cc:504] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:25.114065 22294 leader_election.cc:288] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [CANDIDATE]: Term 1 election: Requested vote from peers a95269497ed247d79b092f11380f675e (127.28.191.1:35435), 4b007b4a6b7c4ea1a962f954434f8d97 (127.28.191.2:35217)
I20260806 02:44:25.114427 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f050308d4a4cf0a6dd2f472eb21cca" candidate_uuid: "fcf236ef707f4bd7adc496f72f05afc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b007b4a6b7c4ea1a962f954434f8d97"
I20260806 02:44:25.114563 22018 raft_consensus.cc:3037] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:25.114580 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f050308d4a4cf0a6dd2f472eb21cca" candidate_uuid: "fcf236ef707f4bd7adc496f72f05afc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95269497ed247d79b092f11380f675e"
I20260806 02:44:25.114718 21840 raft_consensus.cc:3037] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:25.115669 22018 raft_consensus.cc:2454] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcf236ef707f4bd7adc496f72f05afc7 in term 1.
I20260806 02:44:25.115804 21840 raft_consensus.cc:2454] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcf236ef707f4bd7adc496f72f05afc7 in term 1.
I20260806 02:44:25.115974 22110 leader_election.cc:302] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b007b4a6b7c4ea1a962f954434f8d97, fcf236ef707f4bd7adc496f72f05afc7; no voters:
I20260806 02:44:25.116225 22294 raft_consensus.cc:2781] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:25.116418 22294 raft_consensus.cc:686] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 1 LEADER]: Becoming Leader. State: Replica: fcf236ef707f4bd7adc496f72f05afc7, State: Running, Role: LEADER
I20260806 02:44:25.116741 22294 consensus_queue.cc:237] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } }
I20260806 02:44:25.119807 21661 catalog_manager.cc:5495] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcf236ef707f4bd7adc496f72f05afc7 (127.28.191.3). New cstate: current_term: 1 leader_uuid: "fcf236ef707f4bd7adc496f72f05afc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf236ef707f4bd7adc496f72f05afc7" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 } health_report { overall_health: UNKNOWN } } }
I20260806 02:44:25.207243 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.203843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.28.191.3:46245)
0806 02:44:25.203984 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.203987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.204002 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:44:25.204152 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.204155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.204211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.204300 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:25.204537 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.205024 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.205877 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.206114 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.206189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.206502 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:25.206897 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:25.206983 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.207073 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.207124 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:44:25.212848 22018 raft_consensus.cc:1261] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 1 FOLLOWER]: Refusing update from remote peer fcf236ef707f4bd7adc496f72f05afc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:25.212852 21840 raft_consensus.cc:1261] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 1 FOLLOWER]: Refusing update from remote peer fcf236ef707f4bd7adc496f72f05afc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:25.213646 22294 consensus_queue.cc:1035] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95269497ed247d79b092f11380f675e" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:25.213845 22300 consensus_queue.cc:1035] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b007b4a6b7c4ea1a962f954434f8d97" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:25.221801 22305 mvcc.cc:205] Tried to move back new op lower bound from 7315391550304849920 to 7315391549920788480. Current Snapshot: MvccSnapshot[applied={T|T < 7315391550304849920}]
I20260806 02:44:25.222280 22304 mvcc.cc:205] Tried to move back new op lower bound from 7315391550304849920 to 7315391549920788480. Current Snapshot: MvccSnapshot[applied={T|T < 7315391550304849920}]
I20260806 02:44:25.230193 21662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:25.230348 21662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:25.232897 21662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 819e1102d4f34dbe8c488d3e283b1b88: Sending DeleteTablet for 3 replicas of tablet a1f050308d4a4cf0a6dd2f472eb21cca
I20260806 02:44:25.233867 21819 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:41904
I20260806 02:44:25.233944 21991 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:60722
I20260806 02:44:25.234285 22167 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f050308d4a4cf0a6dd2f472eb21cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:25 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:55392
I20260806 02:44:25.234458 22313 tablet_replica.cc:330] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e: stopping tablet replica
I20260806 02:44:25.234846 22313 raft_consensus.cc:2229] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:25.234880 22315 tablet_replica.cc:330] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: stopping tablet replica
I20260806 02:44:25.235176 22313 raft_consensus.cc:2258] T a1f050308d4a4cf0a6dd2f472eb21cca P a95269497ed247d79b092f11380f675e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:25.235265 22315 raft_consensus.cc:2229] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:44:25.235639 22314 tablet_replica.cc:330] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: stopping tablet replica
I20260806 02:44:25.235670 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 21739
I20260806 02:44:25.235687 22315 raft_consensus.cc:2258] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:25.237874 22315 ts_tablet_manager.cc:1905] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:25.241271 22315 ts_tablet_manager.cc:1918] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:25.241406 22315 log.cc:1196] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/a1f050308d4a4cf0a6dd2f472eb21cca
I20260806 02:44:25.241741 22315 ts_tablet_manager.cc:1939] T a1f050308d4a4cf0a6dd2f472eb21cca P fcf236ef707f4bd7adc496f72f05afc7: Deleting consensus metadata
I20260806 02:44:25.243237 21646 catalog_manager.cc:4841] TS fcf236ef707f4bd7adc496f72f05afc7 (127.28.191.3:46245): tablet a1f050308d4a4cf0a6dd2f472eb21cca (table test-table [id=a64f2624e5f647b586ddff21b69aac96]) successfully deleted
I20260806 02:44:25.247033 22314 raft_consensus.cc:2229] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:25.247592 22314 raft_consensus.cc:2258] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:25.250223 22314 ts_tablet_manager.cc:1905] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:25.254042 22314 ts_tablet_manager.cc:1918] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:25.254225 22314 log.cc:1196] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/a1f050308d4a4cf0a6dd2f472eb21cca
I20260806 02:44:25.254698 22314 ts_tablet_manager.cc:1939] T a1f050308d4a4cf0a6dd2f472eb21cca P 4b007b4a6b7c4ea1a962f954434f8d97: Deleting consensus metadata
I20260806 02:44:25.259256 21646 catalog_manager.cc:4841] TS 4b007b4a6b7c4ea1a962f954434f8d97 (127.28.191.2:35217): tablet a1f050308d4a4cf0a6dd2f472eb21cca (table test-table [id=a64f2624e5f647b586ddff21b69aac96]) successfully deleted
W20260806 02:44:25.266844 21649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:35435 (error 108)
I20260806 02:44:25.266919 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 21908
I20260806 02:44:25.269124 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.267737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:35435 (local address 127.0.0.1:41908)
0806 02:44:25.268870 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:25.268951 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:35435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":775,"threads_started":1}
W20260806 02:44:25.269323 21649 catalog_manager.cc:4584] TS a95269497ed247d79b092f11380f675e (127.28.191.1:35435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f050308d4a4cf0a6dd2f472eb21cca: Network error: Client connection negotiation failed: client connection to 127.28.191.1:35435: connect: Connection refused (error 111)
I20260806 02:44:25.278919 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 22084
I20260806 02:44:25.288932 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 21623
2026-08-06T02:44:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2951 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:25Z Disabled control of system clock
I20260806 02:44:25.329818 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42453
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:25.442704 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:25.442936 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:25.442970 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:25.447052 22322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:25.447109 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:25.447125 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:25.447141 22322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:25.447198 22322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:25.451833 22322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:42453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:25.452802 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:25.454272 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:25.462541 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:25.462515 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:25.462498 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:25.462590 22322 server_base.cc:1034] running on GCE node
I20260806 02:44:25.463295 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:25.464337 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:25.465528 22322 hybrid_clock.cc:648] HybridClock initialized: now 1785984265465515 us; error 25 us; skew 500 ppm
I20260806 02:44:25.467408 22322 webserver.cc:460] Webserver started at http://127.28.191.62:42141/ using document root <none> and password file <none>
I20260806 02:44:25.467967 22322 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:25.468070 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:25.468302 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:25.470316 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "8eb46a01351c47a8bc436b5e18318d1a"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "d5633974ece2bda15f2950dad1729ed1"
server_key_iv: "cd58764ad19217bc44f018f81874412c"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.470871 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "8eb46a01351c47a8bc436b5e18318d1a"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "d5633974ece2bda15f2950dad1729ed1"
server_key_iv: "cd58764ad19217bc44f018f81874412c"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.474498 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:25.477154 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.478422 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:44:25.478681 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "8eb46a01351c47a8bc436b5e18318d1a"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "d5633974ece2bda15f2950dad1729ed1"
server_key_iv: "cd58764ad19217bc44f018f81874412c"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.478809 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:25.497591 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:25.498338 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:25.498548 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:25.505842 22322 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:42453
I20260806 02:44:25.505860 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:42453 every 8 connection(s)
I20260806 02:44:25.506788 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:25.510046 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:25.515776 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: Bootstrap starting.
I20260806 02:44:25.515793 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 22322
I20260806 02:44:25.515967 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:25.516302 29436 external_mini_cluster.cc:1428] Setting key ff49135ec6c8978b75037af0fb58b4fb
I20260806 02:44:25.518462 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:25.519208 22407 log.cc:824] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:25.521481 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: No bootstrap required, opened a new log
I20260806 02:44:25.526114 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } }
I20260806 02:44:25.526381 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:25.526445 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb46a01351c47a8bc436b5e18318d1a, State: Initialized, Role: FOLLOWER
I20260806 02:44:25.526809 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.517810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.28.191.62:42453)
0806 02:44:25.518436 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.518450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.518484 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:44:25.519149 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.519165 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.519595 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.519863 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:25.520280 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.521721 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.522660 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.523393 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.523566 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.524412 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:25.524419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:25.524435 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:25.524991 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:25.525110 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:25.525144 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:25.525302 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.525615 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.525876 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":142,"threads_started":1}
I20260806 02:44:25.526934 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } }
I20260806 02:44:25.527101 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:25.527168 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:25.527261 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:25.528292 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } }
I20260806 02:44:25.528633 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb46a01351c47a8bc436b5e18318d1a; no voters:
I20260806 02:44:25.528993 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:25.529098 22414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:25.529314 22414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [term 1 LEADER]: Becoming Leader. State: Replica: 8eb46a01351c47a8bc436b5e18318d1a, State: Running, Role: LEADER
I20260806 02:44:25.529641 22414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } }
I20260806 02:44:25.530202 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:25.531220 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb46a01351c47a8bc436b5e18318d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } } }
I20260806 02:44:25.531361 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:25.531658 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb46a01351c47a8bc436b5e18318d1a. Latest consensus state: current_term: 1 leader_uuid: "8eb46a01351c47a8bc436b5e18318d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb46a01351c47a8bc436b5e18318d1a" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42453 } } }
I20260806 02:44:25.531749 22424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:25.531769 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:25.535346 22424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:25.540658 22424 catalog_manager.cc:1347] Generated new cluster ID: f02ac52382c04d90bfb50ef0fb69ed9c
I20260806 02:44:25.540735 22424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:25.554127 22424 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:25.555011 22424 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:25.562031 22424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: Generated new TSK 0
I20260806 02:44:25.562659 22424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:25.568212 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--builtin_ntp_servers=127.28.191.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:25.692341 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:25.692606 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:25.692646 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:25.696767 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:25.696820 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:25.696874 22436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:25.701732 22436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:25.702806 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:25.704187 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:25.713639 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:25.713711 22436 server_base.cc:1034] running on GCE node
W20260806 02:44:25.713670 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:25.713670 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:25.714293 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:25.714928 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:25.716118 22436 hybrid_clock.cc:648] HybridClock initialized: now 1785984265716090 us; error 39 us; skew 500 ppm
I20260806 02:44:25.718108 22436 webserver.cc:460] Webserver started at http://127.28.191.1:36359/ using document root <none> and password file <none>
I20260806 02:44:25.718719 22436 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:25.718815 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:25.719111 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:25.721321 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "5bfd5489100640598d317e91780bc8db"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "864c5b342f3576f91945642164d7b9d0"
server_key_iv: "5486822254d2077d27ee201c086e9a90"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.721892 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "5bfd5489100640598d317e91780bc8db"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "864c5b342f3576f91945642164d7b9d0"
server_key_iv: "5486822254d2077d27ee201c086e9a90"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.725548 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:44:25.727895 22453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.729120 22436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:25.729269 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "5bfd5489100640598d317e91780bc8db"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "864c5b342f3576f91945642164d7b9d0"
server_key_iv: "5486822254d2077d27ee201c086e9a90"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.729382 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:25.742890 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:25.743530 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:25.743726 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:25.744289 22436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:25.745280 22436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:25.745352 22436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.745424 22436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:25.745468 22436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.754465 22436 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:37503
I20260806 02:44:25.754490 22606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:37503 every 8 connection(s)
I20260806 02:44:25.755347 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:25.756358 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 22436
I20260806 02:44:25.756485 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:25.756843 29436 external_mini_cluster.cc:1428] Setting key ac66711e051f5cd3336f4e0b4efd93fa
I20260806 02:44:25.759123 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--builtin_ntp_servers=127.28.191.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:25.766878 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.757428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:39977 (local address 127.28.191.62:42453)
0806 02:44:25.757578 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.757583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.758820 (+ 1237us) server_negotiation.cc:407] Connection header received
0806 02:44:25.760789 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.760794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.760869 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.760965 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:25.762032 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.762912 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.764768 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.765017 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.765176 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.766419 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:25.766422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:25.766424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:25.766435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:25.766454 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:25.766461 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:25.766520 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.766723 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.766772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:44:25.768148 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.757858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42453 (local address 127.28.191.1:39977)
0806 02:44:25.758664 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:25.758718 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:25.760561 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:25.761335 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:25.761342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:25.761421 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:25.761873 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.761885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.763055 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.763090 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.764626 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.764642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.765253 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.765258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.765477 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.766097 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:25.766101 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:25.766155 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:25.766240 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:25.766554 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:25.766566 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:25.766574 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:44:25.766712 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 02:44:25.767059 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260806 02:44:25.769415 22607 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42453
I20260806 02:44:25.769655 22607 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:25.770140 22607 heartbeater.cc:511] Master 127.28.191.62:42453 requested a full tablet report, sending...
I20260806 02:44:25.771860 22359 ts_manager.cc:194] Registered new tserver with Master: 5bfd5489100640598d317e91780bc8db (127.28.191.1:37503)
I20260806 02:44:25.773629 22359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.1:39977
W20260806 02:44:25.876910 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:25.877215 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:25.877285 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:25.881376 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:25.881436 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:25.881491 22610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:25.886387 22610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:25.887518 22610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:25.888916 22610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:44:25.898890 22610 server_base.cc:1034] running on GCE node
W20260806 02:44:25.898860 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:25.898831 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:25.898839 22618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:25.899570 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:25.900203 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:25.901389 22610 hybrid_clock.cc:648] HybridClock initialized: now 1785984265901376 us; error 27 us; skew 500 ppm
I20260806 02:44:25.903184 22610 webserver.cc:460] Webserver started at http://127.28.191.2:45589/ using document root <none> and password file <none>
I20260806 02:44:25.903765 22610 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:25.903864 22610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:25.904114 22610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:25.906190 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "cb92f4c3b00d435a8ef2b5d487303830"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "3ededb13d949fb6907a1d8ce1d6d01bc"
server_key_iv: "6f63797b14efee85d11ed63d8891adf3"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.906765 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "cb92f4c3b00d435a8ef2b5d487303830"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "3ededb13d949fb6907a1d8ce1d6d01bc"
server_key_iv: "6f63797b14efee85d11ed63d8891adf3"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.910449 22610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:25.912763 22629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.914021 22610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:25.914191 22610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "cb92f4c3b00d435a8ef2b5d487303830"
format_stamp: "Formatted at 2026-08-06 02:44:25 on dist-test-slave-tfpf"
server_key: "3ededb13d949fb6907a1d8ce1d6d01bc"
server_key_iv: "6f63797b14efee85d11ed63d8891adf3"
server_key_version: "encryptionkey@0"
I20260806 02:44:25.914306 22610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:25.937086 22610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:25.937747 22610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:25.937970 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:25.938536 22610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:25.939481 22610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:25.939558 22610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.939625 22610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:25.939673 22610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:25.948434 22610 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:35343
I20260806 02:44:25.948479 22770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:35343 every 8 connection(s)
I20260806 02:44:25.949347 22610 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:25.958019 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 22610
I20260806 02:44:25.958182 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:44:25.958346 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.951100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:55373 (local address 127.28.191.62:42453)
0806 02:44:25.951243 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:25.951247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:25.952292 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 02:44:25.953795 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:25.953798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:25.953845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:25.953920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:25.954792 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.955321 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.956370 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.956575 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.956647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.957745 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:25.957751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:25.957756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:25.957772 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:25.957796 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:25.957806 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:25.957889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:25.958054 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:44:25.958124 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:44:25.958592 29436 external_mini_cluster.cc:1428] Setting key 14f4f139f363d1432d8bf2e437472b96
I20260806 02:44:25.959174 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:25.951487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42453 (local address 127.28.191.2:55373)
0806 02:44:25.952133 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:25.952172 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:25.953604 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:25.954093 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:25.954099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:25.954178 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:25.954656 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.954669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.955430 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.955432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.956264 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:25.956272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:25.956700 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:25.956703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:25.956877 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:25.957469 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:25.957470 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:25.957507 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:25.957573 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:25.957895 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:25.957901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:25.957905 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:44:25.958007 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:44:25.958323 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260806 02:44:25.960553 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--builtin_ntp_servers=127.28.191.20:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:25.960566 22771 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42453
I20260806 02:44:25.960799 22771 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:25.961409 22771 heartbeater.cc:511] Master 127.28.191.62:42453 requested a full tablet report, sending...
I20260806 02:44:25.962513 22359 ts_manager.cc:194] Registered new tserver with Master: cb92f4c3b00d435a8ef2b5d487303830 (127.28.191.2:35343)
I20260806 02:44:25.963104 22359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.2:55373
W20260806 02:44:26.076679 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:26.077021 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:26.077116 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:26.081167 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:26.081243 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:26.081338 22776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:26.086140 22776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:26.087224 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:26.088642 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:26.098883 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:26.098887 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:26.098930 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:26.099359 22776 server_base.cc:1034] running on GCE node
I20260806 02:44:26.099699 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:26.100261 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:26.101418 22776 hybrid_clock.cc:648] HybridClock initialized: now 1785984266101405 us; error 24 us; skew 500 ppm
I20260806 02:44:26.103474 22776 webserver.cc:460] Webserver started at http://127.28.191.3:45841/ using document root <none> and password file <none>
I20260806 02:44:26.104001 22776 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:26.104068 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:26.104234 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:26.106289 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "0c9ba735f8cd4351838acb2f5e4115bb"
format_stamp: "Formatted at 2026-08-06 02:44:26 on dist-test-slave-tfpf"
server_key: "e936778d0218f37dc7dbc4d873f0a414"
server_key_iv: "0862c1b6e98f2669e59090f55ef2b9fe"
server_key_version: "encryptionkey@0"
I20260806 02:44:26.106801 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "0c9ba735f8cd4351838acb2f5e4115bb"
format_stamp: "Formatted at 2026-08-06 02:44:26 on dist-test-slave-tfpf"
server_key: "e936778d0218f37dc7dbc4d873f0a414"
server_key_iv: "0862c1b6e98f2669e59090f55ef2b9fe"
server_key_version: "encryptionkey@0"
I20260806 02:44:26.110309 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:26.112511 22792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:26.113723 22776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:26.113816 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "0c9ba735f8cd4351838acb2f5e4115bb"
format_stamp: "Formatted at 2026-08-06 02:44:26 on dist-test-slave-tfpf"
server_key: "e936778d0218f37dc7dbc4d873f0a414"
server_key_iv: "0862c1b6e98f2669e59090f55ef2b9fe"
server_key_version: "encryptionkey@0"
I20260806 02:44:26.113941 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:26.134150 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:26.134821 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:26.135023 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:26.135624 22776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:26.136564 22776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:26.136610 22776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:26.136675 22776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:26.136708 22776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:26.145169 22776 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:35843
I20260806 02:44:26.145206 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:35843 every 8 connection(s)
I20260806 02:44:26.146023 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:26.148391 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 22776
I20260806 02:44:26.148518 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:26.148809 29436 external_mini_cluster.cc:1428] Setting key c31c5da72832d957edf1eef259da8e3e
I20260806 02:44:26.155349 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.147815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:58273 (local address 127.28.191.62:42453)
0806 02:44:26.147959 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.147963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.149016 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 02:44:26.150795 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.150799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.150851 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.150939 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:26.151812 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.152337 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.153546 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.153770 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.153852 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.154919 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:26.154922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:26.154924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:26.154935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:26.154953 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:26.154960 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:26.155020 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.155194 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.155243 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:44:26.156111 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.148189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:42453 (local address 127.28.191.3:58273)
0806 02:44:26.148868 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.148918 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.150589 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.151149 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.151156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.151236 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:26.151686 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.151699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.152440 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.152442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.153438 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.153448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.153892 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.153895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.154036 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.154676 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:26.154678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:26.154715 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:26.154775 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:26.155036 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:26.155042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:26.155046 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.155158 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.155359 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260806 02:44:26.157222 22942 heartbeater.cc:347] Connected to a master server at 127.28.191.62:42453
I20260806 02:44:26.157418 22942 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:26.157853 22942 heartbeater.cc:511] Master 127.28.191.62:42453 requested a full tablet report, sending...
I20260806 02:44:26.158776 22359 ts_manager.cc:194] Registered new tserver with Master: 0c9ba735f8cd4351838acb2f5e4115bb (127.28.191.3:35843)
I20260806 02:44:26.159339 22359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.3:58273
I20260806 02:44:26.162600 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:26.168308 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.164570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.28.191.62:42453)
0806 02:44:26.164686 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.164689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.164850 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:44:26.164932 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.164934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.164968 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.165032 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:26.165334 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.165794 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.166562 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.166750 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.166821 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.167761 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:26.167764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:26.167767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:26.167779 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:26.167800 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:26.167810 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:26.167886 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.168071 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.168126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:44:26.171708 22359 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:26.176872 22359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:26.191434 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.185330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:35343 (local address 127.0.0.1:41150)
0806 02:44:26.185883 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.185931 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.186073 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.186886 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.186891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.186916 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:26.187088 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.187094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188618 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.188621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.190129 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.190141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.190981 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.190984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.191066 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191100 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.191146 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.191289 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":123,"thread_start_us":65,"threads_started":1}
I20260806 02:44:26.191653 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.186448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.28.191.1:37503)
0806 02:44:26.186891 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.186902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.186923 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:44:26.186991 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.186996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.187219 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.187597 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:26.187705 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188412 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.190261 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.191024 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.191142 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191327 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.191377 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.191469 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":121,"threads_started":1}
I20260806 02:44:26.191756 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.185480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.28.191.2:35343)
0806 02:44:26.186376 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.186390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.186415 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:44:26.186492 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.186499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.186703 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.186904 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:26.187218 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188450 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.190279 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.190851 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.191261 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191432 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.191486 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.191585 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":186,"threads_started":1}
I20260806 02:44:26.191879 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.186280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.28.191.3:35843)
0806 02:44:26.186808 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.186822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.186845 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:44:26.186912 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.186917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.187115 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.187278 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:26.187478 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188564 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.189974 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.190705 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.190816 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191003 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.191630 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.191719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":92,"threads_started":1}
I20260806 02:44:26.192188 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.186038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:35843 (local address 127.0.0.1:58246)
0806 02:44:26.186638 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.186651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.186724 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.187244 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.187247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.187263 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:26.187367 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.187373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188724 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.188728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.189851 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.189860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.190840 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.190844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.190970 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191018 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.191125 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.192090 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":76,"thread_start_us":89,"threads_started":1}
I20260806 02:44:26.192809 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.185475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37503 (local address 127.0.0.1:59242)
0806 02:44:26.186006 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.186020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.186462 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.187364 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.187366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.187381 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:26.187473 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.187478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.188526 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.188531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.190129 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.190149 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.191144 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.191148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.191255 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.191277 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.191343 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.191413 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":222,"thread_start_us":90,"threads_started":1}
I20260806 02:44:26.194928 22692 tablet_service.cc:1467] Processing CreateTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 (DEFAULT_TABLE table=test-table [id=a75ab0160d284be98d37a5f12648d482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:26.194928 22856 tablet_service.cc:1467] Processing CreateTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 (DEFAULT_TABLE table=test-table [id=a75ab0160d284be98d37a5f12648d482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:26.196075 22692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ec2c8353c24601ac66dadaff135bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:26.196075 22856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ec2c8353c24601ac66dadaff135bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:26.199225 22517 tablet_service.cc:1467] Processing CreateTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 (DEFAULT_TABLE table=test-table [id=a75ab0160d284be98d37a5f12648d482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:26.200045 22517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ec2c8353c24601ac66dadaff135bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:26.203480 22963 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Bootstrap starting.
I20260806 02:44:26.203917 22964 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Bootstrap starting.
I20260806 02:44:26.206251 22964 tablet_bootstrap.cc:654] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:26.206816 22963 tablet_bootstrap.cc:654] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:26.207360 22964 log.cc:824] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:26.207686 22963 log.cc:824] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:26.208069 22965 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Bootstrap starting.
I20260806 02:44:26.209283 22964 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: No bootstrap required, opened a new log
I20260806 02:44:26.209487 22964 ts_tablet_manager.cc:1397] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:44:26.209676 22963 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: No bootstrap required, opened a new log
I20260806 02:44:26.209849 22963 ts_tablet_manager.cc:1397] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:44:26.210085 22965 tablet_bootstrap.cc:654] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:26.210896 22965 log.cc:824] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:26.212809 22965 tablet_bootstrap.cc:492] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: No bootstrap required, opened a new log
I20260806 02:44:26.212980 22965 ts_tablet_manager.cc:1397] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:44:26.213001 22964 raft_consensus.cc:348] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.213320 22964 raft_consensus.cc:374] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:26.213388 22964 raft_consensus.cc:729] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ba735f8cd4351838acb2f5e4115bb, State: Initialized, Role: FOLLOWER
I20260806 02:44:26.213858 22964 consensus_queue.cc:260] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.213964 22963 raft_consensus.cc:348] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.214219 22963 raft_consensus.cc:374] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:26.214289 22963 raft_consensus.cc:729] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb92f4c3b00d435a8ef2b5d487303830, State: Initialized, Role: FOLLOWER
I20260806 02:44:26.214699 22963 consensus_queue.cc:260] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.214812 22964 ts_tablet_manager.cc:1428] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:44:26.215183 22942 heartbeater.cc:503] Master 127.28.191.62:42453 was elected leader, sending a full tablet report...
I20260806 02:44:26.215318 22771 heartbeater.cc:503] Master 127.28.191.62:42453 was elected leader, sending a full tablet report...
I20260806 02:44:26.216815 22963 ts_tablet_manager.cc:1428] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:44:26.217491 22965 raft_consensus.cc:348] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.217705 22965 raft_consensus.cc:374] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:26.217772 22965 raft_consensus.cc:729] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfd5489100640598d317e91780bc8db, State: Initialized, Role: FOLLOWER
I20260806 02:44:26.218153 22965 consensus_queue.cc:260] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.218812 22607 heartbeater.cc:503] Master 127.28.191.62:42453 was elected leader, sending a full tablet report...
I20260806 02:44:26.219094 22965 ts_tablet_manager.cc:1428] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:44:26.227768 22972 raft_consensus.cc:482] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:26.227938 22972 raft_consensus.cc:504] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.228915 22972 leader_election.cc:288] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb92f4c3b00d435a8ef2b5d487303830 (127.28.191.2:35343), 5bfd5489100640598d317e91780bc8db (127.28.191.1:37503)
I20260806 02:44:26.232658 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.229122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:35343 (local address 127.28.191.3:60433)
0806 02:44:26.229262 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.229279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.229388 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.229640 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.229643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.229661 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:26.229799 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.229806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.230519 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.230524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.231472 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.231482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.232229 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.232233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.232340 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.232368 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.232440 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.232511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":20}
I20260806 02:44:26.232697 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.229203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60433 (local address 127.28.191.2:35343)
0806 02:44:26.229332 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.229336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.229351 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:44:26.229496 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.229499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.229537 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.229627 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:26.229914 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.230409 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.231596 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.232114 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.232233 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.232283 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.232475 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.232557 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:44:26.233577 22719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ec2c8353c24601ac66dadaff135bf8" candidate_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb92f4c3b00d435a8ef2b5d487303830" is_pre_election: true
I20260806 02:44:26.233856 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.229266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:57907 (local address 127.28.191.1:37503)
0806 02:44:26.229391 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.229395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.229946 (+ 551us) server_negotiation.cc:407] Connection header received
0806 02:44:26.230067 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.230070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.230115 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.230196 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:26.230461 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.231104 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.232606 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.233216 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.233287 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.233343 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.233677 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.233751 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:44:26.233888 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.229529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37503 (local address 127.28.191.3:57907)
0806 02:44:26.229858 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.229869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:26.229980 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:26.230207 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:26.230210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:26.230225 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:26.230353 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.230359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.231212 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.231217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.232209 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:26.232218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.233450 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.233454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:26.233547 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.233565 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:44:26.233626 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:44:26.233680 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":19,"thread_start_us":120,"threads_started":1}
I20260806 02:44:26.233867 22719 raft_consensus.cc:2454] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9ba735f8cd4351838acb2f5e4115bb in term 0.
I20260806 02:44:26.234377 22796 leader_election.cc:302] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ba735f8cd4351838acb2f5e4115bb, cb92f4c3b00d435a8ef2b5d487303830; no voters:
I20260806 02:44:26.234650 22972 raft_consensus.cc:2781] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:26.234612 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ec2c8353c24601ac66dadaff135bf8" candidate_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bfd5489100640598d317e91780bc8db" is_pre_election: true
I20260806 02:44:26.234745 22972 raft_consensus.cc:482] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:44:26.234808 22972 raft_consensus.cc:3037] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:26.234895 22541 raft_consensus.cc:2454] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9ba735f8cd4351838acb2f5e4115bb in term 0.
I20260806 02:44:26.235937 22972 raft_consensus.cc:504] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.236271 22972 leader_election.cc:288] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [CANDIDATE]: Term 1 election: Requested vote from peers cb92f4c3b00d435a8ef2b5d487303830 (127.28.191.2:35343), 5bfd5489100640598d317e91780bc8db (127.28.191.1:37503)
I20260806 02:44:26.236616 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ec2c8353c24601ac66dadaff135bf8" candidate_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bfd5489100640598d317e91780bc8db"
I20260806 02:44:26.236634 22719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ec2c8353c24601ac66dadaff135bf8" candidate_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb92f4c3b00d435a8ef2b5d487303830"
I20260806 02:44:26.236763 22719 raft_consensus.cc:3037] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:26.236758 22541 raft_consensus.cc:3037] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:26.238029 22541 raft_consensus.cc:2454] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9ba735f8cd4351838acb2f5e4115bb in term 1.
I20260806 02:44:26.238029 22719 raft_consensus.cc:2454] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9ba735f8cd4351838acb2f5e4115bb in term 1.
I20260806 02:44:26.238325 22796 leader_election.cc:302] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c9ba735f8cd4351838acb2f5e4115bb, 5bfd5489100640598d317e91780bc8db, cb92f4c3b00d435a8ef2b5d487303830; no voters:
I20260806 02:44:26.238519 22972 raft_consensus.cc:2781] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:26.238680 22972 raft_consensus.cc:686] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 1 LEADER]: Becoming Leader. State: Replica: 0c9ba735f8cd4351838acb2f5e4115bb, State: Running, Role: LEADER
I20260806 02:44:26.238986 22972 consensus_queue.cc:237] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } }
I20260806 02:44:26.242051 22358 catalog_manager.cc:5495] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c9ba735f8cd4351838acb2f5e4115bb (127.28.191.3). New cstate: current_term: 1 leader_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9ba735f8cd4351838acb2f5e4115bb" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35843 } health_report { overall_health: HEALTHY } } }
I20260806 02:44:26.252516 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.249375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.28.191.3:35843)
0806 02:44:26.249499 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:26.249502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:26.249516 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:44:26.249659 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:26.249661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:26.249705 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:26.249786 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:26.249957 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.250405 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.251196 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:26.251457 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:26.251563 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:26.251786 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:26.252169 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:26.252255 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:26.252322 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:44:26.252361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:44:26.256551 22608 tablet.cc:2369] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:26.258872 22541 raft_consensus.cc:1261] T 84ec2c8353c24601ac66dadaff135bf8 P 5bfd5489100640598d317e91780bc8db [term 1 FOLLOWER]: Refusing update from remote peer 0c9ba735f8cd4351838acb2f5e4115bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:26.258872 22719 raft_consensus.cc:1261] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 1 FOLLOWER]: Refusing update from remote peer 0c9ba735f8cd4351838acb2f5e4115bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:26.259734 22978 consensus_queue.cc:1035] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bfd5489100640598d317e91780bc8db" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:26.260044 22972 consensus_queue.cc:1035] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb92f4c3b00d435a8ef2b5d487303830" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:26.268749 22983 mvcc.cc:205] Tried to move back new op lower bound from 7315391554587783168 to 7315391554517475328. Current Snapshot: MvccSnapshot[applied={T|T < 7315391554587783168}]
I20260806 02:44:26.277467 22357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:26.277657 22357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:26.279933 22357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb46a01351c47a8bc436b5e18318d1a: Sending DeleteTablet for 3 replicas of tablet 84ec2c8353c24601ac66dadaff135bf8
I20260806 02:44:26.280655 22692 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:26 UTC) from {username='slave'} at 127.0.0.1:41150
I20260806 02:44:26.280778 22517 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:26 UTC) from {username='slave'} at 127.0.0.1:59242
I20260806 02:44:26.281093 22856 tablet_service.cc:1514] Processing DeleteTablet for tablet 84ec2c8353c24601ac66dadaff135bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:26 UTC) from {username='slave'} at 127.0.0.1:58246
I20260806 02:44:26.281262 22996 tablet_replica.cc:330] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: stopping tablet replica
I20260806 02:44:26.281522 22999 tablet_replica.cc:330] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: stopping tablet replica
I20260806 02:44:26.281635 22996 raft_consensus.cc:2229] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:26.281893 22999 raft_consensus.cc:2229] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:44:26.281939 22996 raft_consensus.cc:2258] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:26.281984 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 22436
I20260806 02:44:26.291512 22999 raft_consensus.cc:2258] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:26.292115 22996 ts_tablet_manager.cc:1905] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:26.292960 22999 ts_tablet_manager.cc:1905] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:26.295284 22996 ts_tablet_manager.cc:1918] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:26.295387 22996 log.cc:1196] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/84ec2c8353c24601ac66dadaff135bf8
I20260806 02:44:26.295758 22996 ts_tablet_manager.cc:1939] T 84ec2c8353c24601ac66dadaff135bf8 P cb92f4c3b00d435a8ef2b5d487303830: Deleting consensus metadata
I20260806 02:44:26.295799 22999 ts_tablet_manager.cc:1918] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:26.295876 22999 log.cc:1196] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/84ec2c8353c24601ac66dadaff135bf8
I20260806 02:44:26.296198 22999 ts_tablet_manager.cc:1939] T 84ec2c8353c24601ac66dadaff135bf8 P 0c9ba735f8cd4351838acb2f5e4115bb: Deleting consensus metadata
I20260806 02:44:26.297245 22343 catalog_manager.cc:4841] TS cb92f4c3b00d435a8ef2b5d487303830 (127.28.191.2:35343): tablet 84ec2c8353c24601ac66dadaff135bf8 (table test-table [id=a75ab0160d284be98d37a5f12648d482]) successfully deleted
I20260806 02:44:26.297336 22340 catalog_manager.cc:4841] TS 0c9ba735f8cd4351838acb2f5e4115bb (127.28.191.3:35843): tablet 84ec2c8353c24601ac66dadaff135bf8 (table test-table [id=a75ab0160d284be98d37a5f12648d482]) successfully deleted
W20260806 02:44:26.303418 22340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:37503 (error 108)
I20260806 02:44:26.303568 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 22610
I20260806 02:44:26.311679 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:26.311279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:37503 (local address 127.0.0.1:59246)
0806 02:44:26.311469 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:26.311562 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 02:44:26.311969 22340 catalog_manager.cc:4584] TS 5bfd5489100640598d317e91780bc8db (127.28.191.1:37503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84ec2c8353c24601ac66dadaff135bf8: Network error: Client connection negotiation failed: client connection to 127.28.191.1:37503: connect: Connection refused (error 111)
I20260806 02:44:26.316269 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 22776
I20260806 02:44:26.327539 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 22322
2026-08-06T02:44:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:26 dist-test-slave-tfpf krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:26 dist-test-slave-tfpf krb5kdc[23004](info): set up 2 sockets
Aug 06 02:44:26 dist-test-slave-tfpf krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260806 02:44:27.552587 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:27 dist-test-slave-tfpf krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:27.683897 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36325
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:27.797204 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:27.797452 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:27.797492 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:27.801441 23020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:27.801491 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:27.801509 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:27.801525 23020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:27.801540 23020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:27.806586 23020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:27.807590 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:27.808516 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:44:27.817094 23020 server_base.cc:1034] running on GCE node
W20260806 02:44:27.817025 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:27.817037 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:27.817021 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:27.817930 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:27.818953 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:27.820125 23020 hybrid_clock.cc:648] HybridClock initialized: now 1785984267820106 us; error 30 us; skew 500 ppm
Aug 06 02:44:27 dist-test-slave-tfpf krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:27.826429 23020 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:27.827421 23020 webserver.cc:460] Webserver started at http://127.28.191.62:45221/ using document root <none> and password file <none>
I20260806 02:44:27.828032 23020 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:27.828135 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:27.828389 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:27.830605 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "c4684624c2c94913a04b88e826751fee"
format_stamp: "Formatted at 2026-08-06 02:44:27 on dist-test-slave-tfpf"
server_key: "1ad0fbb0a00dfbfbf57bd35b13a7169e"
server_key_iv: "8379fcc773c7a5e358329dbe4b200906"
server_key_version: "encryptionkey@0"
I20260806 02:44:27.831225 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "c4684624c2c94913a04b88e826751fee"
format_stamp: "Formatted at 2026-08-06 02:44:27 on dist-test-slave-tfpf"
server_key: "1ad0fbb0a00dfbfbf57bd35b13a7169e"
server_key_iv: "8379fcc773c7a5e358329dbe4b200906"
server_key_version: "encryptionkey@0"
I20260806 02:44:27.835212 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:27.837755 23038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:27.838982 23020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:27.839147 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "c4684624c2c94913a04b88e826751fee"
format_stamp: "Formatted at 2026-08-06 02:44:27 on dist-test-slave-tfpf"
server_key: "1ad0fbb0a00dfbfbf57bd35b13a7169e"
server_key_iv: "8379fcc773c7a5e358329dbe4b200906"
server_key_version: "encryptionkey@0"
I20260806 02:44:27.839268 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:27.853751 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:27.856624 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:27.856860 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:27.863809 23020 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36325
I20260806 02:44:27.863821 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36325 every 8 connection(s)
I20260806 02:44:27.864758 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:27.867523 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:27.870965 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23020
I20260806 02:44:27.871114 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:27.871438 29436 external_mini_cluster.cc:1428] Setting key 30fad19a8a27d1d1df51f971398d3cb4
I20260806 02:44:27.873473 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: Bootstrap starting.
I20260806 02:44:27.875587 23108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:27.876325 23108 log.cc:824] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:27.878165 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: No bootstrap required, opened a new log
Aug 06 02:44:27 dist-test-slave-tfpf krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:27.882369 23108 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } }
I20260806 02:44:27.882583 23108 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:27.882619 23108 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4684624c2c94913a04b88e826751fee, State: Initialized, Role: FOLLOWER
I20260806 02:44:27.883121 23108 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } }
I20260806 02:44:27.883373 23108 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:27.883478 23108 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:27.883597 23108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:27.884516 23108 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } }
I20260806 02:44:27.884898 23108 leader_election.cc:302] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4684624c2c94913a04b88e826751fee; no voters:
I20260806 02:44:27.885316 23108 leader_election.cc:288] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:27.885668 23113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:27.886000 23113 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [term 1 LEADER]: Becoming Leader. State: Replica: c4684624c2c94913a04b88e826751fee, State: Running, Role: LEADER
I20260806 02:44:27.886314 23113 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } }
I20260806 02:44:27.886583 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:27.872868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.28.191.62:36325)
0806 02:44:27.873333 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:27.873349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:27.873383 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:44:27.874039 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:27.874076 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:27.874384 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:27.874713 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:27.875577 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:27.876683 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:27.877617 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:27.878081 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:27.878255 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:27.881099 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:27.881108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:27.881129 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:27.881156 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:27.883105 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:27.883689 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:27.883698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:27.883707 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:27.883845 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:27.884200 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:27.884204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:27.884206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:27.884652 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:27.884840 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:27.885209 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 02:44:27.885554 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
I20260806 02:44:27.886682 23108 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:27.887621 23113 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4684624c2c94913a04b88e826751fee. Latest consensus state: current_term: 1 leader_uuid: "c4684624c2c94913a04b88e826751fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } } }
I20260806 02:44:27.888223 23113 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:27.888561 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4684624c2c94913a04b88e826751fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4684624c2c94913a04b88e826751fee" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36325 } } }
I20260806 02:44:27.888692 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:27.888799 23118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:27.891990 23118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:27.897475 23118 catalog_manager.cc:1347] Generated new cluster ID: 33203bd421f541f19d9205f11ef8d8f0
I20260806 02:44:27.897547 23118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:27.911681 23118 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:27.912971 23118 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:27.923723 23118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: Generated new TSK 0
I20260806 02:44:27.924487 23118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:44:27.959668 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:28.072993 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:28.073261 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:28.073302 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:28.077459 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:28.077517 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:28.077597 23138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:28.082738 23138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:28.083722 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:28.084744 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:28.094389 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.094403 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.094398 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:28.094743 23138 server_base.cc:1034] running on GCE node
I20260806 02:44:28.095165 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:28.095774 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:28.096967 23138 hybrid_clock.cc:648] HybridClock initialized: now 1785984268096941 us; error 37 us; skew 500 ppm
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:28.099733 23138 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:44:28.100591 23138 webserver.cc:460] Webserver started at http://127.28.191.1:35111/ using document root <none> and password file <none>
I20260806 02:44:28.101208 23138 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:28.101306 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:28.101557 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:28.103530 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "26e87e31c51341ab9b9d6d13aed74e24"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "7cd1c8c65b927806bbc53069354a8fcd"
server_key_iv: "b1580ef8159c29aceeb72bb65e322a24"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.104097 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "26e87e31c51341ab9b9d6d13aed74e24"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "7cd1c8c65b927806bbc53069354a8fcd"
server_key_iv: "b1580ef8159c29aceeb72bb65e322a24"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.107757 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:44:28.110131 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.111267 23138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:44:28.111414 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "26e87e31c51341ab9b9d6d13aed74e24"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "7cd1c8c65b927806bbc53069354a8fcd"
server_key_iv: "b1580ef8159c29aceeb72bb65e322a24"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.111526 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:28.143808 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:28.146822 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:28.147043 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:28.147639 23138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:28.148495 23138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:28.148569 23138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.148641 23138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:28.148693 23138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.157497 23138 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:39201
I20260806 02:44:28.157546 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:39201 every 8 connection(s)
I20260806 02:44:28.158444 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:28.166239 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23138
I20260806 02:44:28.166386 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:28.166776 29436 external_mini_cluster.cc:1428] Setting key 56fbe2ec71b8522c91ef1a431f60a5e7
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:28.172356 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.160318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:34123 (local address 127.28.191.62:36325)
0806 02:44:28.160442 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.160446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.161515 (+ 1069us) server_negotiation.cc:407] Connection header received
0806 02:44:28.162400 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.162404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.162460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.162570 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:28.163705 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.164245 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.165463 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.165698 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.165819 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.168620 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:28.168623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:28.168624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:28.168648 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:28.170136 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.170740 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.170745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.170750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.170850 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.171188 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.171193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.171196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.171437 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:28.171550 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.171983 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.172121 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:44:28.172919 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.160659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36325 (local address 127.28.191.1:34123)
0806 02:44:28.161366 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.161409 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.162202 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.162752 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.162758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.163098 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:28.163549 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.163563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.164378 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.164382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.165336 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.165347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.165857 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.165862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.166025 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.166670 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:28.166691 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:28.168462 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:28.170261 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.170268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.170282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.170642 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.170948 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.170950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.170951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.171096 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.171550 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:28.171555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:28.171749 (+ 194us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.171884 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.172134 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":101,"threads_started":1}
I20260806 02:44:28.174145 23306 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36325
I20260806 02:44:28.174401 23306 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:28.174853 23306 heartbeater.cc:511] Master 127.28.191.62:36325 requested a full tablet report, sending...
I20260806 02:44:28.176303 23061 ts_manager.cc:194] Registered new tserver with Master: 26e87e31c51341ab9b9d6d13aed74e24 (127.28.191.1:39201)
I20260806 02:44:28.177523 23061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:34123
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:44:28.196122 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:28.309165 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:28.309461 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:28.309572 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:28.313552 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:28.313627 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:28.313709 23312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:28.319250 23312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:28.320318 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:28.321369 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:28.330905 23322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.330874 23319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.330883 23320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:28.331116 23312 server_base.cc:1034] running on GCE node
I20260806 02:44:28.331506 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:28.332145 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:28.333362 23312 hybrid_clock.cc:648] HybridClock initialized: now 1785984268333341 us; error 30 us; skew 500 ppm
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:28.335919 23312 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:44:28.336752 23312 webserver.cc:460] Webserver started at http://127.28.191.2:37473/ using document root <none> and password file <none>
I20260806 02:44:28.337432 23312 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:28.337529 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:28.337736 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:28.339694 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "759880da0d284632ada77b7074dc93bb"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "a0f44de60ce5e0850cbb14425c710b4a"
server_key_iv: "c35a0a3c6a1377d2cd77601224f1e6ac"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.340252 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "759880da0d284632ada77b7074dc93bb"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "a0f44de60ce5e0850cbb14425c710b4a"
server_key_iv: "c35a0a3c6a1377d2cd77601224f1e6ac"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.344216 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:44:28.346581 23331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.347743 23312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:44:28.347888 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "759880da0d284632ada77b7074dc93bb"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "a0f44de60ce5e0850cbb14425c710b4a"
server_key_iv: "c35a0a3c6a1377d2cd77601224f1e6ac"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.347998 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:28.363889 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:28.366593 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:28.366781 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:28.367340 23312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:28.368139 23312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:28.368211 23312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.368281 23312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:28.368322 23312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.377103 23312 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:41737
I20260806 02:44:28.377125 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:41737 every 8 connection(s)
I20260806 02:44:28.378186 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:28.383154 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23312
I20260806 02:44:28.383303 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:44:28.383684 29436 external_mini_cluster.cc:1428] Setting key 8ade67cc26cfcaaf26913e68765b2160
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:28.393034 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.380189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:34515 (local address 127.28.191.62:36325)
0806 02:44:28.380365 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.380369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.381261 (+ 892us) server_negotiation.cc:407] Connection header received
0806 02:44:28.382103 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.382106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.382150 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.382224 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:28.383607 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.384190 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.385758 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.386136 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.386273 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.389412 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:28.389417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:28.389423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:28.389453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:28.390905 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.391413 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.391416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.391418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.391500 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.391843 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.391846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.391848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.392030 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:28.392141 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.392744 (+ 603us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.392866 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:44:28.393826 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.380441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36325 (local address 127.28.191.2:34515)
0806 02:44:28.381108 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.381164 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.381935 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.382376 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.382383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.382754 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:28.383446 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.383468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.384375 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.384384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.385611 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.385626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.386301 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.386306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.386532 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.387216 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:28.387238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:28.389201 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:28.391024 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.391030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.391044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.391301 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.391597 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.391600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.391602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.391737 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.392157 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:28.392165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:28.392457 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.392686 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.392991 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
I20260806 02:44:28.395157 23477 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36325
I20260806 02:44:28.395390 23477 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:28.395869 23477 heartbeater.cc:511] Master 127.28.191.62:36325 requested a full tablet report, sending...
I20260806 02:44:28.396898 23061 ts_manager.cc:194] Registered new tserver with Master: 759880da0d284632ada77b7074dc93bb (127.28.191.2:41737)
I20260806 02:44:28.397478 23061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:34515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:44:28.412227 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:38525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:28.526253 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:28.526515 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:28.526557 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:28.530555 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:28.530608 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:28.530663 23485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:28.535784 23485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:38525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:28.537016 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:28.538323 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:28.548190 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.548242 23492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:28.548200 23496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:28.548403 23485 server_base.cc:1034] running on GCE node
I20260806 02:44:28.548939 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:28.549618 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:28.550773 23485 hybrid_clock.cc:648] HybridClock initialized: now 1785984268550744 us; error 40 us; skew 500 ppm
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:28.553717 23485 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:44:28.554548 23485 webserver.cc:460] Webserver started at http://127.28.191.3:36877/ using document root <none> and password file <none>
I20260806 02:44:28.555111 23485 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:28.555205 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:28.555435 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:28.557507 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "90f5e6ae63f448eeb5db8b1621e6f990"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "c59978c07f001fa5b846ddeb1ea5bca3"
server_key_iv: "8b6d9bc1d0aa81838f5d4595f8586e33"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.558089 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "90f5e6ae63f448eeb5db8b1621e6f990"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "c59978c07f001fa5b846ddeb1ea5bca3"
server_key_iv: "8b6d9bc1d0aa81838f5d4595f8586e33"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.561841 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:28.564381 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.565615 23485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:28.565761 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "90f5e6ae63f448eeb5db8b1621e6f990"
format_stamp: "Formatted at 2026-08-06 02:44:28 on dist-test-slave-tfpf"
server_key: "c59978c07f001fa5b846ddeb1ea5bca3"
server_key_iv: "8b6d9bc1d0aa81838f5d4595f8586e33"
server_key_version: "encryptionkey@0"
I20260806 02:44:28.565882 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:28.591450 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:28.594353 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:28.594568 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:28.595211 23485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:28.596050 23485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:28.596122 23485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.596194 23485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:28.596237 23485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:28.604934 23485 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:42549
I20260806 02:44:28.604964 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:42549 every 8 connection(s)
I20260806 02:44:28.605862 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:28.609617 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23485
I20260806 02:44:28.609755 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:28.610064 29436 external_mini_cluster.cc:1428] Setting key efb352ea552a358f926cf7c1348f9689
Aug 06 02:44:28 dist-test-slave-tfpf krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:28.619750 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.607636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:51725 (local address 127.28.191.62:36325)
0806 02:44:28.607810 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.607815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.608799 (+ 984us) server_negotiation.cc:407] Connection header received
0806 02:44:28.609847 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.609851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.609908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.610015 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:28.611314 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.611967 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.613233 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.613447 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.613539 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.616085 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:28.616088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:28.616090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:28.616136 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:28.617673 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.618202 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.618207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.618209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.618300 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.618652 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.618656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.618658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.618842 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:28.618952 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.619449 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.619582 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:44:28.620515 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.607995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:36325 (local address 127.28.191.3:51725)
0806 02:44:28.608645 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.608687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.609575 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.610191 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.610197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.610537 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:28.611160 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.611180 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.612099 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.612104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.613125 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.613136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.613584 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.613587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.613722 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.614291 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:28.614311 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:28.615906 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:28.617812 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.617818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.617832 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.618097 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.618406 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:28.618410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:28.618412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:28.618553 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:28.618953 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:28.618961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:28.619210 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.619394 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.619701 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":113,"threads_started":1}
I20260806 02:44:28.621639 23649 heartbeater.cc:347] Connected to a master server at 127.28.191.62:36325
I20260806 02:44:28.621881 23649 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:28.622409 23649 heartbeater.cc:511] Master 127.28.191.62:36325 requested a full tablet report, sending...
I20260806 02:44:28.623469 23060 ts_manager.cc:194] Registered new tserver with Master: 90f5e6ae63f448eeb5db8b1621e6f990 (127.28.191.3:42549)
I20260806 02:44:28.624176 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:28.624250 23060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:51725
I20260806 02:44:28.633927 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.626332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.28.191.62:36325)
0806 02:44:28.626480 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.626484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.626651 (+ 167us) server_negotiation.cc:407] Connection header received
0806 02:44:28.626782 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.626785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.626829 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.626914 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:28.627624 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.628120 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.628955 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.629251 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.629363 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.630860 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:28.630863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:28.630866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:28.630890 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:28.632026 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.632526 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.632529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.632531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.632622 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:28.632952 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:28.632955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:28.632957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:28.633245 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:28.633404 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.633679 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.633769 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:44:28.636751 23060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:28.639683 23060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:28.653888 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.647716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:41737 (local address 127.0.0.1:34240)
0806 02:44:28.648605 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.648654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.648851 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.649593 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.649595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.649610 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:28.649714 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.649721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.651126 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.651129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.652206 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.652217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.653283 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.653410 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.653486 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.653612 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.653749 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260806 02:44:28.653905 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.647851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.28.191.2:41737)
0806 02:44:28.648440 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.648454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.648742 (+ 288us) server_negotiation.cc:407] Connection header received
0806 02:44:28.649028 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.649034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.649256 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.649385 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:28.649811 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.650965 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.652326 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.653164 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.653449 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.653655 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.653749 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":93,"threads_started":1}
I20260806 02:44:28.654346 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.647929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:42549 (local address 127.0.0.1:34066)
0806 02:44:28.648752 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.648766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.648851 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.649427 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.649434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.649452 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:28.649616 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.649624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.650792 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.650797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.652066 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.652079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.653409 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.653525 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.653548 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.653611 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.654258 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":558,"thread_start_us":109,"threads_started":1}
I20260806 02:44:28.654592 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.648544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39201 (local address 127.0.0.1:53394)
0806 02:44:28.649325 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.649337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.649441 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.650122 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.650125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.650144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:28.650236 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.650241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.651531 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.651534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.652876 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.652888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.654218 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.654222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.654323 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.654345 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.654407 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.654470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":43,"thread_start_us":101,"threads_started":1}
I20260806 02:44:28.654600 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.648079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.28.191.3:42549)
0806 02:44:28.648623 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.648636 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.648984 (+ 348us) server_negotiation.cc:407] Connection header received
0806 02:44:28.649080 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.649086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.649255 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.649398 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:28.649747 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.650669 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.652199 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.653014 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653182 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.653349 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.653744 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.653834 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":137,"threads_started":1}
I20260806 02:44:28.655211 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.648954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.28.191.1:39201)
0806 02:44:28.649658 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.649674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.649697 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:44:28.649769 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.649776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.649951 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.650114 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:28.650346 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.651396 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653023 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.653816 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.653926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.654930 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.654978 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.655081 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":120,"threads_started":1}
I20260806 02:44:28.657119 23568 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2595f58af04e339ca11b987f899526 (DEFAULT_TABLE table=test-table [id=5ab1e9dc3baa4443ba1053c3581c7700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:28.657116 23391 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2595f58af04e339ca11b987f899526 (DEFAULT_TABLE table=test-table [id=5ab1e9dc3baa4443ba1053c3581c7700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:28.658032 23568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2595f58af04e339ca11b987f899526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:28.657666 23222 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2595f58af04e339ca11b987f899526 (DEFAULT_TABLE table=test-table [id=5ab1e9dc3baa4443ba1053c3581c7700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:28.658118 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2595f58af04e339ca11b987f899526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:28.658576 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2595f58af04e339ca11b987f899526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:28.665040 23672 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Bootstrap starting.
I20260806 02:44:28.665904 23673 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Bootstrap starting.
I20260806 02:44:28.665904 23674 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Bootstrap starting.
I20260806 02:44:28.667011 23672 tablet_bootstrap.cc:654] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:28.668020 23673 tablet_bootstrap.cc:654] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:28.668059 23672 log.cc:824] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:28.668463 23674 tablet_bootstrap.cc:654] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:28.669320 23673 log.cc:824] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:28.669320 23674 log.cc:824] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:28.670360 23672 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: No bootstrap required, opened a new log
I20260806 02:44:28.670585 23672 ts_tablet_manager.cc:1397] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:44:28.671283 23673 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: No bootstrap required, opened a new log
I20260806 02:44:28.671415 23674 tablet_bootstrap.cc:492] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: No bootstrap required, opened a new log
I20260806 02:44:28.671459 23673 ts_tablet_manager.cc:1397] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:44:28.671587 23674 ts_tablet_manager.cc:1397] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:44:28.674419 23674 raft_consensus.cc:348] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.674443 23672 raft_consensus.cc:348] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.674671 23674 raft_consensus.cc:374] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:28.674710 23672 raft_consensus.cc:374] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:28.674729 23674 raft_consensus.cc:729] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759880da0d284632ada77b7074dc93bb, State: Initialized, Role: FOLLOWER
I20260806 02:44:28.674760 23672 raft_consensus.cc:729] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f5e6ae63f448eeb5db8b1621e6f990, State: Initialized, Role: FOLLOWER
I20260806 02:44:28.675170 23674 consensus_queue.cc:260] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.675086 23673 raft_consensus.cc:348] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.675184 23672 consensus_queue.cc:260] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.675310 23673 raft_consensus.cc:374] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:28.675374 23673 raft_consensus.cc:729] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e87e31c51341ab9b9d6d13aed74e24, State: Initialized, Role: FOLLOWER
I20260806 02:44:28.675803 23477 heartbeater.cc:503] Master 127.28.191.62:36325 was elected leader, sending a full tablet report...
I20260806 02:44:28.675784 23673 consensus_queue.cc:260] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.676137 23672 ts_tablet_manager.cc:1428] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:44:28.676143 23674 ts_tablet_manager.cc:1428] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:28.676430 23649 heartbeater.cc:503] Master 127.28.191.62:36325 was elected leader, sending a full tablet report...
I20260806 02:44:28.676571 23673 ts_tablet_manager.cc:1428] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 02:44:28.676995 23306 heartbeater.cc:503] Master 127.28.191.62:36325 was elected leader, sending a full tablet report...
W20260806 02:44:28.856868 23650 tablet.cc:2369] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:44:28.879545 23479 tablet.cc:2369] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:28.900956 23678 raft_consensus.cc:482] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:28.901216 23678 raft_consensus.cc:504] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.902325 23678 leader_election.cc:288] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f5e6ae63f448eeb5db8b1621e6f990 (127.28.191.3:42549), 26e87e31c51341ab9b9d6d13aed74e24 (127.28.191.1:39201)
I20260806 02:44:28.907788 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.902732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:34083 (local address 127.28.191.3:42549)
0806 02:44:28.902965 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.902968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.903091 (+ 123us) server_negotiation.cc:407] Connection header received
0806 02:44:28.903202 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.903206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.903257 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.903373 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:28.903797 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.904598 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.906401 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.907015 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.907145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.907199 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.907582 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.907655 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260806 02:44:28.907788 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.902720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:47113 (local address 127.28.191.1:39201)
0806 02:44:28.902863 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.902870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.903139 (+ 269us) server_negotiation.cc:407] Connection header received
0806 02:44:28.903221 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.903224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.903270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.903375 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:28.903797 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.904598 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.906401 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.907015 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.907145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.907199 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.907589 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.907655 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:28.907850 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.902584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39201 (local address 127.28.191.2:47113)
0806 02:44:28.903026 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.903040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.903139 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.903408 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.903411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.903455 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:28.903674 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.903682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.904717 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.904721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.906236 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.906248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.907155 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.907159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.907422 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.907472 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.907550 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.907655 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":345,"spinlock_wait_cycles":34560,"thread_start_us":71,"threads_started":1}
I20260806 02:44:28.907850 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.902584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:42549 (local address 127.28.191.2:34083)
0806 02:44:28.902954 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.902970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:28.903102 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:28.903408 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:28.903411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:28.903430 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:28.903674 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.903682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.904717 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.904721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.906236 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:28.906248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.907155 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.907159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:28.907383 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.907467 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:44:28.907550 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:44:28.907655 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":285,"spinlock_wait_cycles":9344,"thread_start_us":105,"threads_started":1}
I20260806 02:44:28.908578 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2595f58af04e339ca11b987f899526" candidate_uuid: "759880da0d284632ada77b7074dc93bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" is_pre_election: true
I20260806 02:44:28.908696 23248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2595f58af04e339ca11b987f899526" candidate_uuid: "759880da0d284632ada77b7074dc93bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e87e31c51341ab9b9d6d13aed74e24" is_pre_election: true
I20260806 02:44:28.908928 23591 raft_consensus.cc:2454] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759880da0d284632ada77b7074dc93bb in term 0.
I20260806 02:44:28.909041 23248 raft_consensus.cc:2454] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 759880da0d284632ada77b7074dc93bb in term 0.
I20260806 02:44:28.909437 23332 leader_election.cc:302] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759880da0d284632ada77b7074dc93bb, 90f5e6ae63f448eeb5db8b1621e6f990; no voters:
I20260806 02:44:28.909744 23678 raft_consensus.cc:2781] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:28.909842 23678 raft_consensus.cc:482] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 02:44:28.909860 23307 tablet.cc:2369] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:28.909906 23678 raft_consensus.cc:3037] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:28.911074 23678 raft_consensus.cc:504] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.911401 23678 leader_election.cc:288] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [CANDIDATE]: Term 1 election: Requested vote from peers 90f5e6ae63f448eeb5db8b1621e6f990 (127.28.191.3:42549), 26e87e31c51341ab9b9d6d13aed74e24 (127.28.191.1:39201)
I20260806 02:44:28.911747 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2595f58af04e339ca11b987f899526" candidate_uuid: "759880da0d284632ada77b7074dc93bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f5e6ae63f448eeb5db8b1621e6f990"
I20260806 02:44:28.911783 23248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2595f58af04e339ca11b987f899526" candidate_uuid: "759880da0d284632ada77b7074dc93bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e87e31c51341ab9b9d6d13aed74e24"
I20260806 02:44:28.911845 23591 raft_consensus.cc:3037] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:28.911922 23248 raft_consensus.cc:3037] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:28.912869 23591 raft_consensus.cc:2454] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759880da0d284632ada77b7074dc93bb in term 1.
I20260806 02:44:28.913033 23248 raft_consensus.cc:2454] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 759880da0d284632ada77b7074dc93bb in term 1.
I20260806 02:44:28.913189 23332 leader_election.cc:302] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759880da0d284632ada77b7074dc93bb, 90f5e6ae63f448eeb5db8b1621e6f990; no voters:
I20260806 02:44:28.913424 23678 raft_consensus.cc:2781] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:28.913620 23678 raft_consensus.cc:686] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 1 LEADER]: Becoming Leader. State: Replica: 759880da0d284632ada77b7074dc93bb, State: Running, Role: LEADER
I20260806 02:44:28.913961 23678 consensus_queue.cc:237] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } }
I20260806 02:44:28.917210 23059 catalog_manager.cc:5495] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 759880da0d284632ada77b7074dc93bb (127.28.191.2). New cstate: current_term: 1 leader_uuid: "759880da0d284632ada77b7074dc93bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "759880da0d284632ada77b7074dc93bb" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 41737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 } health_report { overall_health: UNKNOWN } } }
I20260806 02:44:28.945525 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.942055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.28.191.2:41737)
0806 02:44:28.942179 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:28.942182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:28.942196 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:44:28.942307 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:28.942309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:28.942360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:28.942446 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:28.942668 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.943214 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.944097 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:28.944296 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:28.944369 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:28.944689 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:28.945184 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:28.945300 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:28.945389 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:44:28.945422 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:44:28.951159 23591 raft_consensus.cc:1261] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 1 FOLLOWER]: Refusing update from remote peer 759880da0d284632ada77b7074dc93bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:28.951167 23248 raft_consensus.cc:1261] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 1 FOLLOWER]: Refusing update from remote peer 759880da0d284632ada77b7074dc93bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:28.951807 23678 consensus_queue.cc:1035] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e87e31c51341ab9b9d6d13aed74e24" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:28.952051 23683 consensus_queue.cc:1035] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f5e6ae63f448eeb5db8b1621e6f990" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:28.969830 23059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:28.970191 23059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:28.973686 23059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4684624c2c94913a04b88e826751fee: Sending DeleteTablet for 3 replicas of tablet 3b2595f58af04e339ca11b987f899526
I20260806 02:44:28.974444 23391 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2595f58af04e339ca11b987f899526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34240
I20260806 02:44:28.974748 23222 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2595f58af04e339ca11b987f899526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:53394
I20260806 02:44:28.975003 23706 tablet_replica.cc:330] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: stopping tablet replica
I20260806 02:44:28.975337 23706 raft_consensus.cc:2229] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:44:28.975359 23568 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2595f58af04e339ca11b987f899526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:28 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:34066
I20260806 02:44:28.975828 23706 raft_consensus.cc:2258] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:28.976058 23708 tablet_replica.cc:330] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: stopping tablet replica
I20260806 02:44:28.976317 23708 raft_consensus.cc:2229] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:28.976626 23708 raft_consensus.cc:2258] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:28.975843 23707 tablet_replica.cc:330] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24: stopping tablet replica
I20260806 02:44:28.976830 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23138
I20260806 02:44:28.977123 23707 raft_consensus.cc:2229] T 3b2595f58af04e339ca11b987f899526 P 26e87e31c51341ab9b9d6d13aed74e24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:28.977950 23706 ts_tablet_manager.cc:1905] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:28.980337 23706 ts_tablet_manager.cc:1918] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:28.980473 23706 log.cc:1196] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/3b2595f58af04e339ca11b987f899526
I20260806 02:44:28.980808 23706 ts_tablet_manager.cc:1939] T 3b2595f58af04e339ca11b987f899526 P 759880da0d284632ada77b7074dc93bb: Deleting consensus metadata
I20260806 02:44:28.981925 23044 catalog_manager.cc:4841] TS 759880da0d284632ada77b7074dc93bb (127.28.191.2:41737): tablet 3b2595f58af04e339ca11b987f899526 (table test-table [id=5ab1e9dc3baa4443ba1053c3581c7700]) successfully deleted
I20260806 02:44:28.987972 23708 ts_tablet_manager.cc:1905] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:28.990680 23708 ts_tablet_manager.cc:1918] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:28.990808 23708 log.cc:1196] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/3b2595f58af04e339ca11b987f899526
I20260806 02:44:28.991119 23708 ts_tablet_manager.cc:1939] T 3b2595f58af04e339ca11b987f899526 P 90f5e6ae63f448eeb5db8b1621e6f990: Deleting consensus metadata
I20260806 02:44:28.992390 23041 catalog_manager.cc:4841] TS 90f5e6ae63f448eeb5db8b1621e6f990 (127.28.191.3:42549): tablet 3b2595f58af04e339ca11b987f899526 (table test-table [id=5ab1e9dc3baa4443ba1053c3581c7700]) successfully deleted
W20260806 02:44:28.993515 23045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:39201 (error 108)
I20260806 02:44:28.993837 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23312
I20260806 02:44:28.998799 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:28.998474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:39201 (local address 127.0.0.1:53404)
0806 02:44:28.998643 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:28.998716 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:39201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:44:28.998988 23045 catalog_manager.cc:4584] TS 26e87e31c51341ab9b9d6d13aed74e24 (127.28.191.1:39201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b2595f58af04e339ca11b987f899526: Network error: Client connection negotiation failed: client connection to 127.28.191.1:39201: connect: Connection refused (error 111)
I20260806 02:44:29.004182 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23485
I20260806 02:44:29.014153 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23020
2026-08-06T02:44:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2688 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:29Z Disabled control of system clock
I20260806 02:44:29.051283 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46261
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:29.161907 23714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:29.162146 23714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:29.162178 23714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:29.166080 23714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:29.166126 23714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:29.166141 23714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:29.166157 23714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:29.166170 23714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:29.170727 23714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:29.171700 23714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:29.172986 23714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:29.181617 23723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:29.181679 23714 server_base.cc:1034] running on GCE node
W20260806 02:44:29.181665 23722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.181826 23727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:29.182404 23714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:29.183517 23714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:29.184693 23714 hybrid_clock.cc:648] HybridClock initialized: now 1785984269184679 us; error 25 us; skew 500 ppm
I20260806 02:44:29.186559 23714 webserver.cc:460] Webserver started at http://127.28.191.62:46209/ using document root <none> and password file <none>
I20260806 02:44:29.187107 23714 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:29.187167 23714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:29.187377 23714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:29.189473 23714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "45f26e23cff14925919db565737a11fd"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "5daafa9b06a6fc67aff41ea908df8bba"
server_key_iv: "00e0cb996893b69e92efb8c0f49828f3"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.190032 23714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "45f26e23cff14925919db565737a11fd"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "5daafa9b06a6fc67aff41ea908df8bba"
server_key_iv: "00e0cb996893b69e92efb8c0f49828f3"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.193655 23714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:44:29.195947 23735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.197253 23714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:29.197376 23714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "45f26e23cff14925919db565737a11fd"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "5daafa9b06a6fc67aff41ea908df8bba"
server_key_iv: "00e0cb996893b69e92efb8c0f49828f3"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.197475 23714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:29.208071 23714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:29.208690 23714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:29.208879 23714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:29.215807 23714 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:46261
I20260806 02:44:29.215823 23799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:46261 every 8 connection(s)
I20260806 02:44:29.216777 23714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:29.220005 23804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:29.225752 23804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: Bootstrap starting.
I20260806 02:44:29.227075 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23714
I20260806 02:44:29.227217 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:29.227569 29436 external_mini_cluster.cc:1428] Setting key 7780d0b12c8cd64d85de348322f5a190
I20260806 02:44:29.228855 23804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:29.229732 23804 log.cc:824] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:29.231983 23804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: No bootstrap required, opened a new log
I20260806 02:44:29.236246 23804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } }
I20260806 02:44:29.236451 23804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:29.236526 23804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f26e23cff14925919db565737a11fd, State: Initialized, Role: FOLLOWER
I20260806 02:44:29.237021 23804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } }
I20260806 02:44:29.237164 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.228859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.28.191.62:46261)
0806 02:44:29.229558 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.229575 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.229611 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:44:29.230417 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.230443 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.230838 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.231109 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:29.231478 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.232494 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.233743 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.234205 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.234324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.235223 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:29.235228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:29.235240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:29.235599 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:29.235650 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:29.235679 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:29.235756 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.235919 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.236206 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":198,"threads_started":1}
I20260806 02:44:29.237198 23804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:29.237308 23804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:29.237404 23804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:29.238343 23804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } }
I20260806 02:44:29.238692 23804 leader_election.cc:302] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f26e23cff14925919db565737a11fd; no voters:
I20260806 02:44:29.239003 23804 leader_election.cc:288] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:29.239110 23809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:29.239351 23809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [term 1 LEADER]: Becoming Leader. State: Replica: 45f26e23cff14925919db565737a11fd, State: Running, Role: LEADER
I20260806 02:44:29.239643 23809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } }
I20260806 02:44:29.240175 23804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:29.241374 23810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f26e23cff14925919db565737a11fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } } }
I20260806 02:44:29.241504 23810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:29.241791 23811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f26e23cff14925919db565737a11fd. Latest consensus state: current_term: 1 leader_uuid: "45f26e23cff14925919db565737a11fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f26e23cff14925919db565737a11fd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46261 } } }
I20260806 02:44:29.241878 23811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd [sys.catalog]: This master's current role is: LEADER
W20260806 02:44:29.245112 23830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:44:29.245208 23830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:44:29.245283 23819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:29.246002 23819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:29.250944 23819 catalog_manager.cc:1347] Generated new cluster ID: c98abbe331af4161afa15bbb9a82f0f5
I20260806 02:44:29.251016 23819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:29.261413 23819 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:29.262652 23819 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:29.270212 23819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: Generated new TSK 0
I20260806 02:44:29.271057 23819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:29.278592 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--builtin_ntp_servers=127.28.191.20:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:29.393575 23833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:29.393811 23833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:29.393847 23833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:29.398088 23833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:29.398146 23833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:29.398211 23833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:29.403192 23833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:29.404238 23833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:29.405695 23833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:29.415465 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.415473 23843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.415491 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:29.415709 23833 server_base.cc:1034] running on GCE node
I20260806 02:44:29.416278 23833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:29.416918 23833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:29.418112 23833 hybrid_clock.cc:648] HybridClock initialized: now 1785984269418095 us; error 33 us; skew 500 ppm
I20260806 02:44:29.419895 23833 webserver.cc:460] Webserver started at http://127.28.191.1:45693/ using document root <none> and password file <none>
I20260806 02:44:29.420498 23833 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:29.420590 23833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:29.420827 23833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:29.422930 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "f5e2f54e6a14455d9e6ed8c269103d8d"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "09ca53cca923d0516bf068dba6202b70"
server_key_iv: "dbb2a1b955ff06aa5959460c2bd077fb"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.423516 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "f5e2f54e6a14455d9e6ed8c269103d8d"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "09ca53cca923d0516bf068dba6202b70"
server_key_iv: "dbb2a1b955ff06aa5959460c2bd077fb"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.427140 23833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:44:29.429518 23850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.430639 23833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:29.430776 23833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "f5e2f54e6a14455d9e6ed8c269103d8d"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "09ca53cca923d0516bf068dba6202b70"
server_key_iv: "dbb2a1b955ff06aa5959460c2bd077fb"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.430891 23833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:29.451460 23833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:29.452097 23833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:29.452299 23833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:29.452870 23833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:29.453840 23833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:29.453913 23833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.453979 23833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:29.454030 23833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.463203 23833 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:45489
I20260806 02:44:29.463261 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:45489 every 8 connection(s)
I20260806 02:44:29.464236 23833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:29.466022 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23833
I20260806 02:44:29.466202 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:29.466581 29436 external_mini_cluster.cc:1428] Setting key 23e079e68309fa7b41da42f18c0a015a
I20260806 02:44:29.468600 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--builtin_ntp_servers=127.28.191.20:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:29.474431 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.466357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:40165 (local address 127.28.191.62:46261)
0806 02:44:29.466536 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.466541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.467738 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 02:44:29.469405 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.469409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.469475 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.469588 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:29.470512 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.471075 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.472286 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.472521 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.472634 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.473893 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:29.473896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:29.473898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:29.473912 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:29.473930 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:29.473937 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:29.474034 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.474205 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.474278 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:44:29.475391 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.466806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46261 (local address 127.28.191.1:40165)
0806 02:44:29.467570 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.467624 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.469176 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.469774 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.469780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.469865 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:29.470359 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.470371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.471223 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.471226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.472150 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.472159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.472681 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.472685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.472859 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.473620 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:29.473621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:29.473663 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:29.473729 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:29.474037 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:29.474050 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:29.474056 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.474192 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.474511 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260806 02:44:29.476713 23989 heartbeater.cc:347] Connected to a master server at 127.28.191.62:46261
I20260806 02:44:29.476980 23989 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:29.477587 23989 heartbeater.cc:511] Master 127.28.191.62:46261 requested a full tablet report, sending...
I20260806 02:44:29.479305 23757 ts_manager.cc:194] Registered new tserver with Master: f5e2f54e6a14455d9e6ed8c269103d8d (127.28.191.1:45489)
I20260806 02:44:29.481091 23757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.1:40165
W20260806 02:44:29.584843 23993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:29.585166 23993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:29.585227 23993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:29.589254 23993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:29.589298 23993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:29.589351 23993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:29.593906 23993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:29.594846 23993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:29.596225 23993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:29.605676 24000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.605696 24002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.605786 24004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:29.605863 23993 server_base.cc:1034] running on GCE node
I20260806 02:44:29.606329 23993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:29.606917 23993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:29.608098 23993 hybrid_clock.cc:648] HybridClock initialized: now 1785984269608067 us; error 41 us; skew 500 ppm
I20260806 02:44:29.609964 23993 webserver.cc:460] Webserver started at http://127.28.191.2:43869/ using document root <none> and password file <none>
I20260806 02:44:29.610532 23993 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:29.610597 23993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:29.610826 23993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:29.612802 23993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "a5cbe5c73e3ec29621b2fdc5cb605898"
server_key_iv: "51aca272b8b7a656f3537492f3288ae7"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.613425 23993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "a5cbe5c73e3ec29621b2fdc5cb605898"
server_key_iv: "51aca272b8b7a656f3537492f3288ae7"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.616999 23993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:44:29.619354 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.620527 23993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:29.620658 23993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "a5cbe5c73e3ec29621b2fdc5cb605898"
server_key_iv: "51aca272b8b7a656f3537492f3288ae7"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.620771 23993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:29.636929 23993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:29.637634 23993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:29.637835 23993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:29.638418 23993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:29.639353 23993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:29.639426 23993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.639497 23993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:29.639549 23993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.648416 23993 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:36049
I20260806 02:44:29.648453 24151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:36049 every 8 connection(s)
I20260806 02:44:29.649374 23993 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:29.656370 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 23993
I20260806 02:44:29.656491 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:44:29.656791 29436 external_mini_cluster.cc:1428] Setting key 8fe1cfed1414e8bc0b98d7efe14a72b2
I20260806 02:44:29.658918 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--builtin_ntp_servers=127.28.191.20:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:29.659452 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.651147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:49061 (local address 127.28.191.62:46261)
0806 02:44:29.651296 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.651300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.652335 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 02:44:29.654026 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.654028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.654078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.654154 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:29.655057 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.655588 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.656959 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.657243 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.657370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.658723 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:29.658758 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:29.658766 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:29.658782 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:29.658807 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:29.658847 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:29.658945 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.659180 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.659251 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:44:29.661007 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.651562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46261 (local address 127.28.191.2:49061)
0806 02:44:29.652186 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.652224 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.653798 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.654317 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.654324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.654407 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:29.654908 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.654921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.655711 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.655714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.656801 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.656812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.657410 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.657415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.657601 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.658322 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:29.658324 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:29.658382 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:29.658475 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:29.658994 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:29.659005 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:29.659013 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.659628 (+ 615us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.659918 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260806 02:44:29.662186 24153 heartbeater.cc:347] Connected to a master server at 127.28.191.62:46261
I20260806 02:44:29.662460 24153 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:29.663107 24153 heartbeater.cc:511] Master 127.28.191.62:46261 requested a full tablet report, sending...
I20260806 02:44:29.664235 23757 ts_manager.cc:194] Registered new tserver with Master: a9aac5bea1f348cfa5b5afb9d885c1ce (127.28.191.2:36049)
I20260806 02:44:29.664850 23757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.2:49061
W20260806 02:44:29.777462 24156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:29.777698 24156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:29.777761 24156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:29.781926 24156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:29.781978 24156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:29.782083 24156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:29.786778 24156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:29.787921 24156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:29.789363 24156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:29.799108 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.799117 24164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:29.799124 24166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:29.799212 24156 server_base.cc:1034] running on GCE node
I20260806 02:44:29.799747 24156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:29.800442 24156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:29.801632 24156 hybrid_clock.cc:648] HybridClock initialized: now 1785984269801613 us; error 35 us; skew 500 ppm
I20260806 02:44:29.803468 24156 webserver.cc:460] Webserver started at http://127.28.191.3:44047/ using document root <none> and password file <none>
I20260806 02:44:29.804049 24156 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:29.804142 24156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:29.804351 24156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:29.806424 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "6caad4f22eb440f68c8d3d59464d95ed"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "709d09dbd96ec800a363edfc0b77ff62"
server_key_iv: "9703d60e1a369fd8604362dad64adbe4"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.807001 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "6caad4f22eb440f68c8d3d59464d95ed"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "709d09dbd96ec800a363edfc0b77ff62"
server_key_iv: "9703d60e1a369fd8604362dad64adbe4"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.810580 24156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:44:29.812898 24172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.814110 24156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:29.814246 24156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "6caad4f22eb440f68c8d3d59464d95ed"
format_stamp: "Formatted at 2026-08-06 02:44:29 on dist-test-slave-tfpf"
server_key: "709d09dbd96ec800a363edfc0b77ff62"
server_key_iv: "9703d60e1a369fd8604362dad64adbe4"
server_key_version: "encryptionkey@0"
I20260806 02:44:29.814359 24156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:29.832581 24156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:29.833382 24156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:29.833597 24156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:29.834287 24156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:29.835315 24156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:29.835383 24156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.835448 24156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:29.835497 24156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:29.844445 24156 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:37935
I20260806 02:44:29.844520 24317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:37935 every 8 connection(s)
I20260806 02:44:29.845402 24156 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:29.847219 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 24156
I20260806 02:44:29.847352 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:29.847651 29436 external_mini_cluster.cc:1428] Setting key 5ab723f1f344e22a8949c7d6215dd548
I20260806 02:44:29.854441 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.847274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:49305 (local address 127.28.191.62:46261)
0806 02:44:29.847412 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.847417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.848554 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 02:44:29.850059 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.850062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.850107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.850174 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:29.850996 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.851530 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.852648 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.852867 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.852942 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.854009 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:29.854012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:29.854014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:29.854026 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:29.854054 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:29.854062 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:29.854122 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.854279 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.854330 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:44:29.855312 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.847665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46261 (local address 127.28.191.3:49305)
0806 02:44:29.848408 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.848458 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.849866 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.850330 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.850336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.850421 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:29.850878 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.850890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.851662 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.851665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.852536 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.852546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.852991 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.852993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.853174 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.853743 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:29.853745 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:29.853786 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:29.853852 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:29.854145 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:29.854152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:29.854157 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.854262 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.854479 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260806 02:44:29.856431 24318 heartbeater.cc:347] Connected to a master server at 127.28.191.62:46261
I20260806 02:44:29.856684 24318 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:29.857276 24318 heartbeater.cc:511] Master 127.28.191.62:46261 requested a full tablet report, sending...
I20260806 02:44:29.858305 23757 ts_manager.cc:194] Registered new tserver with Master: 6caad4f22eb440f68c8d3d59464d95ed (127.28.191.3:37935)
I20260806 02:44:29.858865 23757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.3:49305
I20260806 02:44:29.861490 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:29.866940 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.863494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.28.191.62:46261)
0806 02:44:29.863609 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.863612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.863751 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:44:29.863842 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.863844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.863880 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.863946 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:29.864171 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.864647 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.865470 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.865663 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.865735 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.866569 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:29.866571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:29.866573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:29.866584 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:29.866598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:29.866605 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:29.866693 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.866817 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.866848 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:44:29.869746 23757 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:29.872628 23757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:29.886525 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.880909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:36049 (local address 127.0.0.1:43898)
0806 02:44:29.882027 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.882040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.882120 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.882625 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.882628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.882646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:29.882740 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.882747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883691 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.883698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.884955 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.884965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.885979 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.886094 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886132 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.886239 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.886354 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":39,"thread_start_us":83,"threads_started":1}
I20260806 02:44:29.886735 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.881322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.28.191.2:36049)
0806 02:44:29.881763 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.881779 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.882218 (+ 439us) server_negotiation.cc:407] Connection header received
0806 02:44:29.882297 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.882302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.882492 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.882636 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:29.882864 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883578 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885101 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.885842 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885959 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886148 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.886358 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.886466 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":141,"threads_started":1}
I20260806 02:44:29.886747 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.880785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.28.191.3:37935)
0806 02:44:29.881217 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.881230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.881480 (+ 250us) server_negotiation.cc:407] Connection header received
0806 02:44:29.881706 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.881712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.881917 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.882065 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:29.882451 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883579 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885358 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.885867 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885993 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886180 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.886491 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.886586 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":116,"threads_started":1}
I20260806 02:44:29.886888 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.880329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:45489 (local address 127.0.0.1:42702)
0806 02:44:29.881282 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.881329 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.881468 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.882077 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.882083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.882107 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:29.882241 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.882248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883719 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.883722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.885211 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.885217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.886228 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.886231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.886603 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886626 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.886701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.886770 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":436,"thread_start_us":84,"threads_started":1}
I20260806 02:44:29.887240 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.880420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37935 (local address 127.0.0.1:58696)
0806 02:44:29.881383 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.881396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.881550 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.882251 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.882255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.882270 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:29.882355 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.882360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883719 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.883722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.885203 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.885215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.886246 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.886250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.886366 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886386 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.886459 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.886515 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":141,"thread_start_us":101,"threads_started":1}
I20260806 02:44:29.887332 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.880451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.28.191.1:45489)
0806 02:44:29.881111 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.881125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.881617 (+ 492us) server_negotiation.cc:407] Connection header received
0806 02:44:29.881706 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.881711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.881933 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.882097 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:29.882378 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.883585 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.885341 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.886116 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.886220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.886388 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.887072 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.887175 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":134,"threads_started":1}
I20260806 02:44:29.889686 24074 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e17dbe7f542e290eb56ac461139ca (DEFAULT_TABLE table=test-table [id=5d83470c05f14786a2113fa7ae5ead49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:29.889959 23908 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e17dbe7f542e290eb56ac461139ca (DEFAULT_TABLE table=test-table [id=5d83470c05f14786a2113fa7ae5ead49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:29.890681 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e17dbe7f542e290eb56ac461139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:29.890946 23908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e17dbe7f542e290eb56ac461139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:29.890856 24236 tablet_service.cc:1467] Processing CreateTablet for tablet fb9e17dbe7f542e290eb56ac461139ca (DEFAULT_TABLE table=test-table [id=5d83470c05f14786a2113fa7ae5ead49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:29.891769 24236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9e17dbe7f542e290eb56ac461139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:29.896450 24340 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Bootstrap starting.
I20260806 02:44:29.898137 24341 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Bootstrap starting.
I20260806 02:44:29.899224 24340 tablet_bootstrap.cc:654] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:29.899268 24343 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Bootstrap starting.
I20260806 02:44:29.900101 24340 log.cc:824] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:29.900178 24341 tablet_bootstrap.cc:654] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:29.900980 24341 log.cc:824] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:29.901156 24343 tablet_bootstrap.cc:654] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:29.901952 24343 log.cc:824] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:29.902285 24340 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: No bootstrap required, opened a new log
I20260806 02:44:29.902477 24340 ts_tablet_manager.cc:1397] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:44:29.903028 24341 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: No bootstrap required, opened a new log
I20260806 02:44:29.903216 24341 ts_tablet_manager.cc:1397] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260806 02:44:29.903883 24343 tablet_bootstrap.cc:492] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: No bootstrap required, opened a new log
I20260806 02:44:29.904090 24343 ts_tablet_manager.cc:1397] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 02:44:29.906785 24340 raft_consensus.cc:348] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.907050 24340 raft_consensus.cc:374] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:29.907111 24340 raft_consensus.cc:729] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9aac5bea1f348cfa5b5afb9d885c1ce, State: Initialized, Role: FOLLOWER
I20260806 02:44:29.906960 24343 raft_consensus.cc:348] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.907234 24343 raft_consensus.cc:374] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:29.907292 24343 raft_consensus.cc:729] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caad4f22eb440f68c8d3d59464d95ed, State: Initialized, Role: FOLLOWER
I20260806 02:44:29.907276 24341 raft_consensus.cc:348] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.907500 24341 raft_consensus.cc:374] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:29.907557 24341 raft_consensus.cc:729] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e2f54e6a14455d9e6ed8c269103d8d, State: Initialized, Role: FOLLOWER
I20260806 02:44:29.907596 24340 consensus_queue.cc:260] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.907709 24343 consensus_queue.cc:260] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.907974 24341 consensus_queue.cc:260] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.908406 24318 heartbeater.cc:503] Master 127.28.191.62:46261 was elected leader, sending a full tablet report...
I20260806 02:44:29.908551 24340 ts_tablet_manager.cc:1428] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:44:29.908846 24341 ts_tablet_manager.cc:1428] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260806 02:44:29.908876 24153 heartbeater.cc:503] Master 127.28.191.62:46261 was elected leader, sending a full tablet report...
I20260806 02:44:29.909152 23989 heartbeater.cc:503] Master 127.28.191.62:46261 was elected leader, sending a full tablet report...
I20260806 02:44:29.910810 24343 ts_tablet_manager.cc:1428] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
W20260806 02:44:29.965497 23991 tablet.cc:2369] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:29.981249 24347 raft_consensus.cc:482] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:29.981506 24347 raft_consensus.cc:504] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.982923 24347 leader_election.cc:288] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e2f54e6a14455d9e6ed8c269103d8d (127.28.191.1:45489), 6caad4f22eb440f68c8d3d59464d95ed (127.28.191.3:37935)
I20260806 02:44:29.986773 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.983139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:45489 (local address 127.28.191.2:40481)
0806 02:44:29.983282 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.983298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.983384 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.983705 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.983708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.983730 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:29.983881 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.983889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.984736 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.984741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.985616 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.985627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.986297 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.986303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.986466 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.986493 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.986563 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.986616 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":25}
I20260806 02:44:29.986816 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.983228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:40481 (local address 127.28.191.1:45489)
0806 02:44:29.983359 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.983363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.983379 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:44:29.983518 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.983521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.983575 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.983659 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:29.984001 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.984629 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.985744 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.986164 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.986256 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.986295 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.986606 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.986687 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:44:29.986881 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.983343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57745 (local address 127.28.191.3:37935)
0806 02:44:29.983476 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:29.983482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:29.983704 (+ 222us) server_negotiation.cc:407] Connection header received
0806 02:44:29.983838 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:29.983845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:29.983897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:29.983968 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:29.984245 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.984734 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.985837 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.986255 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.986378 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.986434 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:29.986690 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:44:29.986759 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:44:29.986919 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:29.983159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:37935 (local address 127.28.191.2:57745)
0806 02:44:29.983613 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:29.983629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:29.983720 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:29.983999 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:29.984002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:29.984019 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:29.984135 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.984141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.984843 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.984845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.985690 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:29.985704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:29.986399 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:29.986402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:29.986526 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:29.986572 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:44:29.986651 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:44:29.986711 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":34,"thread_start_us":113,"threads_started":1}
I20260806 02:44:29.987600 24264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e17dbe7f542e290eb56ac461139ca" candidate_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caad4f22eb440f68c8d3d59464d95ed" is_pre_election: true
I20260806 02:44:29.987852 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e17dbe7f542e290eb56ac461139ca" candidate_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" is_pre_election: true
I20260806 02:44:29.987943 24264 raft_consensus.cc:2454] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9aac5bea1f348cfa5b5afb9d885c1ce in term 0.
I20260806 02:44:29.988134 23933 raft_consensus.cc:2454] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9aac5bea1f348cfa5b5afb9d885c1ce in term 0.
I20260806 02:44:29.988488 24013 leader_election.cc:302] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caad4f22eb440f68c8d3d59464d95ed, a9aac5bea1f348cfa5b5afb9d885c1ce; no voters:
I20260806 02:44:29.988777 24347 raft_consensus.cc:2781] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:29.988864 24347 raft_consensus.cc:482] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:44:29.988909 24347 raft_consensus.cc:3037] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:29.990008 24347 raft_consensus.cc:504] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.990347 24347 leader_election.cc:288] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [CANDIDATE]: Term 1 election: Requested vote from peers f5e2f54e6a14455d9e6ed8c269103d8d (127.28.191.1:45489), 6caad4f22eb440f68c8d3d59464d95ed (127.28.191.3:37935)
I20260806 02:44:29.990705 24264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e17dbe7f542e290eb56ac461139ca" candidate_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caad4f22eb440f68c8d3d59464d95ed"
I20260806 02:44:29.990736 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9e17dbe7f542e290eb56ac461139ca" candidate_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d"
I20260806 02:44:29.990816 24264 raft_consensus.cc:3037] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:29.990877 23933 raft_consensus.cc:3037] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:29.991915 24264 raft_consensus.cc:2454] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9aac5bea1f348cfa5b5afb9d885c1ce in term 1.
I20260806 02:44:29.991925 23933 raft_consensus.cc:2454] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9aac5bea1f348cfa5b5afb9d885c1ce in term 1.
I20260806 02:44:29.992244 24013 leader_election.cc:302] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caad4f22eb440f68c8d3d59464d95ed, a9aac5bea1f348cfa5b5afb9d885c1ce; no voters:
I20260806 02:44:29.992432 24347 raft_consensus.cc:2781] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:29.992668 24347 raft_consensus.cc:686] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 1 LEADER]: Becoming Leader. State: Replica: a9aac5bea1f348cfa5b5afb9d885c1ce, State: Running, Role: LEADER
I20260806 02:44:29.993014 24347 consensus_queue.cc:237] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } }
I20260806 02:44:29.996290 23757 catalog_manager.cc:5495] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a9aac5bea1f348cfa5b5afb9d885c1ce (127.28.191.2). New cstate: current_term: 1 leader_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9aac5bea1f348cfa5b5afb9d885c1ce" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 36049 } health_report { overall_health: HEALTHY } } }
I20260806 02:44:30.014660 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:30.011125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.28.191.2:36049)
0806 02:44:30.011238 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:30.011242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:30.011256 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:44:30.011361 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:30.011363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:30.011418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:30.011509 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:30.011733 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:30.012251 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:30.013354 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:30.013569 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:30.013649 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:30.013908 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:30.014299 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:30.014408 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:30.014486 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:44:30.014529 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 02:44:30.020727 24264 raft_consensus.cc:1261] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 1 FOLLOWER]: Refusing update from remote peer a9aac5bea1f348cfa5b5afb9d885c1ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:30.020757 23933 raft_consensus.cc:1261] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 1 FOLLOWER]: Refusing update from remote peer a9aac5bea1f348cfa5b5afb9d885c1ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:30.021523 24351 consensus_queue.cc:1035] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caad4f22eb440f68c8d3d59464d95ed" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:30.021708 24347 consensus_queue.cc:1035] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e2f54e6a14455d9e6ed8c269103d8d" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:30.029593 24356 mvcc.cc:205] Tried to move back new op lower bound from 7315391569997750272 to 7315391569894445056. Current Snapshot: MvccSnapshot[applied={T|T < 7315391569997750272}]
I20260806 02:44:30.031180 24361 mvcc.cc:205] Tried to move back new op lower bound from 7315391569997750272 to 7315391569894445056. Current Snapshot: MvccSnapshot[applied={T|T < 7315391569997750272}]
I20260806 02:44:30.039441 23757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:30.039626 23757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:30.042210 23757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45f26e23cff14925919db565737a11fd: Sending DeleteTablet for 3 replicas of tablet fb9e17dbe7f542e290eb56ac461139ca
I20260806 02:44:30.042968 23908 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e17dbe7f542e290eb56ac461139ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:30 UTC) from {username='slave'} at 127.0.0.1:42702
I20260806 02:44:30.042959 24236 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e17dbe7f542e290eb56ac461139ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:30 UTC) from {username='slave'} at 127.0.0.1:58696
I20260806 02:44:30.043479 24371 tablet_replica.cc:330] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: stopping tablet replica
I20260806 02:44:30.043462 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9e17dbe7f542e290eb56ac461139ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:30 UTC) from {username='slave'} at 127.0.0.1:43898
I20260806 02:44:30.043808 24371 raft_consensus.cc:2229] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:30.043980 24374 tablet_replica.cc:330] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: stopping tablet replica
I20260806 02:44:30.044129 24371 raft_consensus.cc:2258] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:30.044504 24373 tablet_replica.cc:330] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d: stopping tablet replica
I20260806 02:44:30.044813 24373 raft_consensus.cc:2229] T fb9e17dbe7f542e290eb56ac461139ca P f5e2f54e6a14455d9e6ed8c269103d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:30.044915 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23833
I20260806 02:44:30.049002 24371 ts_tablet_manager.cc:1905] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:30.052188 24371 ts_tablet_manager.cc:1918] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:30.052278 24374 raft_consensus.cc:2229] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:44:30.052708 24374 raft_consensus.cc:2258] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:30.054265 24371 log.cc:1196] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/fb9e17dbe7f542e290eb56ac461139ca
I20260806 02:44:30.054657 24371 ts_tablet_manager.cc:1939] T fb9e17dbe7f542e290eb56ac461139ca P 6caad4f22eb440f68c8d3d59464d95ed: Deleting consensus metadata
I20260806 02:44:30.056023 23737 catalog_manager.cc:4841] TS 6caad4f22eb440f68c8d3d59464d95ed (127.28.191.3:37935): tablet fb9e17dbe7f542e290eb56ac461139ca (table test-table [id=5d83470c05f14786a2113fa7ae5ead49]) successfully deleted
I20260806 02:44:30.056857 24374 ts_tablet_manager.cc:1905] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:30.060357 24374 ts_tablet_manager.cc:1918] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:30.060469 24374 log.cc:1196] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/fb9e17dbe7f542e290eb56ac461139ca
I20260806 02:44:30.060837 24374 ts_tablet_manager.cc:1939] T fb9e17dbe7f542e290eb56ac461139ca P a9aac5bea1f348cfa5b5afb9d885c1ce: Deleting consensus metadata
I20260806 02:44:30.062016 23736 catalog_manager.cc:4841] TS a9aac5bea1f348cfa5b5afb9d885c1ce (127.28.191.2:36049): tablet fb9e17dbe7f542e290eb56ac461139ca (table test-table [id=5d83470c05f14786a2113fa7ae5ead49]) successfully deleted
W20260806 02:44:30.065055 23739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:45489 (error 108)
I20260806 02:44:30.065227 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23993
I20260806 02:44:30.073099 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:30.072758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:45489 (local address 127.0.0.1:42712)
0806 02:44:30.072874 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:30.072954 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 02:44:30.074174 23739 catalog_manager.cc:4584] TS f5e2f54e6a14455d9e6ed8c269103d8d (127.28.191.1:45489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb9e17dbe7f542e290eb56ac461139ca: Network error: Client connection negotiation failed: client connection to 127.28.191.1:45489: connect: Connection refused (error 111)
I20260806 02:44:30.079483 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 24156
I20260806 02:44:30.089278 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 23714
2026-08-06T02:44:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:30 dist-test-slave-tfpf krb5kdc[24379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:30 dist-test-slave-tfpf krb5kdc[24379](info): set up 2 sockets
Aug 06 02:44:30 dist-test-slave-tfpf krb5kdc[24379](info): commencing operation
krb5kdc: starting...
W20260806 02:44:31.314949 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:31 dist-test-slave-tfpf krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:31.444202 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40183
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:31.557511 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:31.557812 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:31.557894 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:31.562036 24394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:31.562109 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:31.562147 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:31.562203 24394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:31.562238 24394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:31.567647 24394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:31.568761 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:31.569907 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:31.578634 24401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:31.578722 24405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:31.578825 24394 server_base.cc:1034] running on GCE node
W20260806 02:44:31.578660 24402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:31.579541 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:31.580576 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:31.581764 24394 hybrid_clock.cc:648] HybridClock initialized: now 1785984271581733 us; error 45 us; skew 500 ppm
Aug 06 02:44:31 dist-test-slave-tfpf krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:31.584540 24394 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:31.585500 24394 webserver.cc:460] Webserver started at http://127.28.191.62:44103/ using document root <none> and password file <none>
I20260806 02:44:31.586071 24394 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:31.586163 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:31.586403 24394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:31.588439 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "e810a9920dac4166b87554ef52d49335"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "6101ee83996956f3cf3ab4a0690b667f"
server_key_iv: "f6811de599d206f2140f6dba9ef05065"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.589021 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "e810a9920dac4166b87554ef52d49335"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "6101ee83996956f3cf3ab4a0690b667f"
server_key_iv: "f6811de599d206f2140f6dba9ef05065"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.592746 24394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:44:31.595242 24412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:31.596382 24394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:31.596520 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "e810a9920dac4166b87554ef52d49335"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "6101ee83996956f3cf3ab4a0690b667f"
server_key_iv: "f6811de599d206f2140f6dba9ef05065"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.596643 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:31.639329 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:31.642567 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:31.642776 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:31.650374 24394 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:40183
I20260806 02:44:31.650388 24483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:40183 every 8 connection(s)
I20260806 02:44:31.651364 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:31.654137 24484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:31.659467 24484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: Bootstrap starting.
I20260806 02:44:31.661271 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 24394
I20260806 02:44:31.661425 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:31.661700 29436 external_mini_cluster.cc:1428] Setting key 4b2bc4a9b3437cd9e5109e8a43214c55
I20260806 02:44:31.661949 24484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:31.662815 24484 log.cc:824] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:31.665006 24484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: No bootstrap required, opened a new log
I20260806 02:44:31.669359 24484 raft_consensus.cc:348] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } }
I20260806 02:44:31.669631 24484 raft_consensus.cc:374] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:31.669718 24484 raft_consensus.cc:729] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e810a9920dac4166b87554ef52d49335, State: Initialized, Role: FOLLOWER
I20260806 02:44:31.670203 24484 consensus_queue.cc:260] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } }
I20260806 02:44:31.670331 24484 raft_consensus.cc:388] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:31.670418 24484 raft_consensus.cc:482] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:31.670523 24484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:44:31 dist-test-slave-tfpf krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:31.671483 24484 raft_consensus.cc:504] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } }
I20260806 02:44:31.671851 24484 leader_election.cc:302] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e810a9920dac4166b87554ef52d49335; no voters:
I20260806 02:44:31.672279 24484 leader_election.cc:288] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:31.672385 24489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:31.672640 24489 raft_consensus.cc:686] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [term 1 LEADER]: Becoming Leader. State: Replica: e810a9920dac4166b87554ef52d49335, State: Running, Role: LEADER
I20260806 02:44:31.672943 24489 consensus_queue.cc:237] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } }
I20260806 02:44:31.673573 24484 sys_catalog.cc:564] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:31.674618 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e810a9920dac4166b87554ef52d49335. Latest consensus state: current_term: 1 leader_uuid: "e810a9920dac4166b87554ef52d49335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } } }
I20260806 02:44:31.674748 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:31.675087 24490 sys_catalog.cc:455] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e810a9920dac4166b87554ef52d49335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e810a9920dac4166b87554ef52d49335" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40183 } } }
I20260806 02:44:31.675212 24490 sys_catalog.cc:458] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:31.677606 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:31.662825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.28.191.62:40183)
0806 02:44:31.663283 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:31.663304 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:31.663338 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:44:31.664116 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:31.664139 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:31.664513 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:31.664871 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:31.665750 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.667240 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.668088 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.668782 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.668944 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:31.671593 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:31.671599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:31.671613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:31.671642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:31.673846 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:31.674431 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:31.674439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:31.674447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:31.674609 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:31.675105 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:31.675111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:31.675116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:31.675614 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:31.675837 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:31.676499 (+ 662us) server_negotiation.cc:299] Negotiation successful
0806 02:44:31.676840 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
W20260806 02:44:31.678743 24515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:44:31.678822 24515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:44:31.678905 24505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:31.679749 24505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:31.684777 24505 catalog_manager.cc:1347] Generated new cluster ID: 97600041b5764acb85c7f9a5e374ee45
I20260806 02:44:31.684835 24505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:31.717649 24505 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:31.718713 24505 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:31.725759 24505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: Generated new TSK 0
I20260806 02:44:31.726552 24505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.191.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
Entry for principal kudu/127.28.191.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab.
I20260806 02:44:31.760360 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:31.874958 24520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:31.875212 24520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:31.875255 24520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:31.879318 24520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:31.879379 24520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:31.879436 24520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:31.884572 24520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.1.keytab
--principal=kudu/127.28.191.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:31.885650 24520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:31.886624 24520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:31.896063 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:31.896054 24529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:31.896054 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:31.896396 24520 server_base.cc:1034] running on GCE node
I20260806 02:44:31.896907 24520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:31.897720 24520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:31.898936 24520 hybrid_clock.cc:648] HybridClock initialized: now 1785984271898917 us; error 31 us; skew 500 ppm
Aug 06 02:44:31 dist-test-slave-tfpf krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:31.901885 24520 init.cc:377] Logged in from keytab as kudu/127.28.191.1@KRBTEST.COM (short username kudu)
I20260806 02:44:31.902808 24520 webserver.cc:460] Webserver started at http://127.28.191.1:38303/ using document root <none> and password file <none>
I20260806 02:44:31.903394 24520 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:31.903491 24520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:31.903725 24520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:31.905800 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "56087ca1fb0f480289fa3c744a3f3522"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "a90918e91cad06fd4ab390a2707367f1"
server_key_iv: "33cfb3968fca097cbcd428dfbdedb7f9"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.906388 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "56087ca1fb0f480289fa3c744a3f3522"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "a90918e91cad06fd4ab390a2707367f1"
server_key_iv: "33cfb3968fca097cbcd428dfbdedb7f9"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.910331 24520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:44:31.912802 24541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:31.914036 24520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:31.914181 24520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "56087ca1fb0f480289fa3c744a3f3522"
format_stamp: "Formatted at 2026-08-06 02:44:31 on dist-test-slave-tfpf"
server_key: "a90918e91cad06fd4ab390a2707367f1"
server_key_iv: "33cfb3968fca097cbcd428dfbdedb7f9"
server_key_version: "encryptionkey@0"
I20260806 02:44:31.914295 24520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:31.939723 24520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:31.942883 24520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:31.943104 24520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:31.943722 24520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:31.944572 24520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:31.944644 24520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:31.944717 24520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:31.944769 24520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:31.953605 24520 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:45291
I20260806 02:44:31.953651 24684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:45291 every 8 connection(s)
I20260806 02:44:31.954476 24520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:31.957610 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 24520
I20260806 02:44:31.957759 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:31.958066 29436 external_mini_cluster.cc:1428] Setting key 832332c336872cd76099ba885a594ddb
Aug 06 02:44:31 dist-test-slave-tfpf krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984271, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.1@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.2@KRBTEST.COM; defaulting to no policy
I20260806 02:44:31.968490 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:31.956280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:48245 (local address 127.28.191.62:40183)
0806 02:44:31.956416 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:31.956420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:31.957264 (+ 844us) server_negotiation.cc:407] Connection header received
0806 02:44:31.958842 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:31.958846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:31.958903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:31.959009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:31.960365 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.960912 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.962095 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.962324 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.962432 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:31.964839 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:31.964849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:31.964852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:31.964873 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:31.966482 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:31.966967 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:31.966972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:31.966973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:31.967088 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:31.967425 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:31.967430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:31.967432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:31.967611 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:31.967718 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:31.968185 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 02:44:31.968306 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":56}
I20260806 02:44:31.969331 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:31.956619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:40183 (local address 127.28.191.1:48245)
0806 02:44:31.957118 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:31.957168 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:31.958617 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:31.959194 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:31.959202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:31.959714 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:31.960186 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:31.960218 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.961038 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.961042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:31.961961 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:31.961971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:31.962473 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:31.962476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:31.962644 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:31.963216 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:31.963231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:31.964672 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:31.966600 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:31.966605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:31.966618 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:31.966864 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:31.967182 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:31.967187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:31.967189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:31.967327 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:31.967719 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:31.967726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:31.967958 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 02:44:31.968143 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:44:31.968443 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260806 02:44:31.970618 24686 heartbeater.cc:347] Connected to a master server at 127.28.191.62:40183
I20260806 02:44:31.970880 24686 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.2@KRBTEST.COM" created.
I20260806 02:44:31.971338 24686 heartbeater.cc:511] Master 127.28.191.62:40183 requested a full tablet report, sending...
I20260806 02:44:31.972993 24435 ts_manager.cc:194] Registered new tserver with Master: 56087ca1fb0f480289fa3c744a3f3522 (127.28.191.1:45291)
I20260806 02:44:31.974709 24435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.1@KRBTEST.COM'} at 127.28.191.1:48245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
Entry for principal kudu/127.28.191.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab.
I20260806 02:44:31.986151 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:32.100140 24693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:32.100390 24693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:32.100467 24693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:32.104364 24693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:32.104447 24693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:32.104528 24693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:32.109792 24693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.2.keytab
--principal=kudu/127.28.191.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:32.110755 24693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:32.111735 24693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:32.121405 24702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:32.121459 24699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:32.121443 24700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:32.121632 24693 server_base.cc:1034] running on GCE node
I20260806 02:44:32.122249 24693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:32.122906 24693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:32.124104 24693 hybrid_clock.cc:648] HybridClock initialized: now 1785984272124082 us; error 34 us; skew 500 ppm
Aug 06 02:44:32 dist-test-slave-tfpf krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:32.126932 24693 init.cc:377] Logged in from keytab as kudu/127.28.191.2@KRBTEST.COM (short username kudu)
I20260806 02:44:32.127751 24693 webserver.cc:460] Webserver started at http://127.28.191.2:40141/ using document root <none> and password file <none>
I20260806 02:44:32.128278 24693 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:32.128335 24693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:32.128505 24693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:32.130607 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "acaf85c5b0e44294851c3dcfc54ed27e"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "7b07615a22f9a016aa03e82ed8bfcb6e"
server_key_iv: "97048fbd925671c444c20743f2a88255"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.131120 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "acaf85c5b0e44294851c3dcfc54ed27e"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "7b07615a22f9a016aa03e82ed8bfcb6e"
server_key_iv: "97048fbd925671c444c20743f2a88255"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.134894 24693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:32.137324 24715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.138533 24693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:32.138633 24693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "acaf85c5b0e44294851c3dcfc54ed27e"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "7b07615a22f9a016aa03e82ed8bfcb6e"
server_key_iv: "97048fbd925671c444c20743f2a88255"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.138765 24693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:32.150740 24693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:32.153523 24693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:32.153723 24693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:32.154287 24693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:32.155112 24693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:32.155177 24693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.155215 24693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:32.155279 24693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.164052 24693 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:40727
I20260806 02:44:32.164075 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:40727 every 8 connection(s)
I20260806 02:44:32.164884 24693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:32.173597 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 24693
I20260806 02:44:32.173789 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
Aug 06 02:44:32 dist-test-slave-tfpf krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.2@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:32.174197 29436 external_mini_cluster.cc:1428] Setting key 512d4b7008d38a3c8029c204f295e144
I20260806 02:44:32.179090 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.166755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:44353 (local address 127.28.191.62:40183)
0806 02:44:32.166885 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.166888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.167888 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 02:44:32.168794 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.168797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.168859 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.168976 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:32.170078 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.170599 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.171693 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.171912 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.172022 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.174839 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:32.174842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:32.174844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:32.174906 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:32.176731 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.177383 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.177389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.177393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.177481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.177892 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.177898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.177900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.178133 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:32.178241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.178876 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.178975 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:44:32.180145 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.167044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:40183 (local address 127.28.191.2:44353)
0806 02:44:32.167732 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.167771 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.168591 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.169186 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.169193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.169509 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:32.169935 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.169948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.170728 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.170737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.171568 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.171578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.172065 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.172069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.172223 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.172832 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:32.172847 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:32.174654 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:32.176857 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:32.176867 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:32.176884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:32.177286 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:32.177585 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:32.177591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:32.177596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:32.177782 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:32.178254 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:32.178262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:32.178589 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.178791 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.179126 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":103,"threads_started":1}
I20260806 02:44:32.181318 24866 heartbeater.cc:347] Connected to a master server at 127.28.191.62:40183
I20260806 02:44:32.181587 24866 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:32.182053 24866 heartbeater.cc:511] Master 127.28.191.62:40183 requested a full tablet report, sending...
I20260806 02:44:32.183022 24435 ts_manager.cc:194] Registered new tserver with Master: acaf85c5b0e44294851c3dcfc54ed27e (127.28.191.2:40727)
I20260806 02:44:32.183615 24435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.2@KRBTEST.COM'} at 127.28.191.2:44353
WARNING: no policy specified for kudu/127.28.191.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
Entry for principal kudu/127.28.191.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab.
I20260806 02:44:32.203707 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:32.318905 24873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:32.319155 24873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:32.319196 24873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:32.323277 24873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:32.323338 24873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:32.323423 24873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:32.328661 24873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.3.keytab
--principal=kudu/127.28.191.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:32.329779 24873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:32.330834 24873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:32.340735 24883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:32.340729 24878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:32.340986 24873 server_base.cc:1034] running on GCE node
W20260806 02:44:32.340729 24881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:32.341661 24873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:32.342351 24873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:32.343545 24873 hybrid_clock.cc:648] HybridClock initialized: now 1785984272343505 us; error 53 us; skew 500 ppm
Aug 06 02:44:32 dist-test-slave-tfpf krb5kdc[24379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:32.346359 24873 init.cc:377] Logged in from keytab as kudu/127.28.191.3@KRBTEST.COM (short username kudu)
I20260806 02:44:32.347224 24873 webserver.cc:460] Webserver started at http://127.28.191.3:35657/ using document root <none> and password file <none>
I20260806 02:44:32.347802 24873 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:32.347894 24873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:32.348133 24873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:32.350226 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "4172844ca9c24dde9c81a12e5ec61657"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "89afb0671dccf830b1de1f251a006f5f"
server_key_iv: "3dd200ce2248c0222d23c151c33bf1fa"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.350828 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "4172844ca9c24dde9c81a12e5ec61657"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "89afb0671dccf830b1de1f251a006f5f"
server_key_iv: "3dd200ce2248c0222d23c151c33bf1fa"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.354463 24873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:32.356897 24892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.358098 24873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:32.358230 24873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "4172844ca9c24dde9c81a12e5ec61657"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "89afb0671dccf830b1de1f251a006f5f"
server_key_iv: "3dd200ce2248c0222d23c151c33bf1fa"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.358340 24873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:32.370935 24873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:32.374333 24873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:32.374603 24873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:32.375192 24873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:32.376130 24873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:32.376199 24873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.376274 24873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:32.376317 24873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.385129 24873 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:35117
I20260806 02:44:32.385169 25037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:35117 every 8 connection(s)
I20260806 02:44:32.386049 24873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:32.391306 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 24873
I20260806 02:44:32.391465 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:32.391760 29436 external_mini_cluster.cc:1428] Setting key a3859a4d37e6d21a9bf4350f302a4575
Aug 06 02:44:32 dist-test-slave-tfpf krb5kdc[24379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984272, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.3@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:32.399557 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.387877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:60233 (local address 127.28.191.62:40183)
0806 02:44:32.388012 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.388016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.389005 (+ 989us) server_negotiation.cc:407] Connection header received
0806 02:44:32.389908 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.389911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.389971 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.390082 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:32.391217 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.391929 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.393082 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.393324 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.393440 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.396379 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:32.396382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:32.396384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:32.396430 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:32.397809 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.398268 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.398270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.398272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.398344 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.398630 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.398632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.398634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.398793 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:32.398878 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.399327 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.399417 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:44:32.400280 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.388241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:40183 (local address 127.28.191.3:60233)
0806 02:44:32.388858 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.388898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.389700 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.390250 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.390256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.390608 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:32.391064 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.391077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.392053 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.392059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.392920 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.392930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.393478 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.393483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.393662 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.394398 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:44:32.394414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:32.396231 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:32.397912 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:32.397917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:32.397928 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:32.398183 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:32.398422 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:44:32.398424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:44:32.398425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:44:32.398552 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:44:32.398889 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:32.398894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:32.399113 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.399278 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.399506 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":104,"threads_started":1}
I20260806 02:44:32.401589 25038 heartbeater.cc:347] Connected to a master server at 127.28.191.62:40183
I20260806 02:44:32.401823 25038 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:32.402289 25038 heartbeater.cc:511] Master 127.28.191.62:40183 requested a full tablet report, sending...
I20260806 02:44:32.403297 24435 ts_manager.cc:194] Registered new tserver with Master: 4172844ca9c24dde9c81a12e5ec61657 (127.28.191.3:35117)
I20260806 02:44:32.403855 24435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.191.3@KRBTEST.COM'} at 127.28.191.3:60233
I20260806 02:44:32.405635 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:32.414320 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.407638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.28.191.62:40183)
0806 02:44:32.407771 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.407775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.407927 (+ 152us) server_negotiation.cc:407] Connection header received
0806 02:44:32.408005 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.408007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.408041 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.408108 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:32.408770 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.409301 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.410063 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.410251 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.410321 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.411514 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:32.411517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:32.411519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:32.411539 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:32.412782 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.413231 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.413233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.413234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.413305 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:32.413594 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:32.413596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:32.413597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:32.413789 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:32.413892 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.414063 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.414149 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:44:32.416952 24435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:32.420068 24435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:32.434470 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.428385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:35117 (local address 127.0.0.1:39668)
0806 02:44:32.428911 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.428924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.429016 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.430259 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.430262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.430274 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:32.430350 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.430359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431422 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.431428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.433082 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.433094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.434093 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.434096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.434194 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.434215 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.434288 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.434342 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":142,"thread_start_us":91,"threads_started":1}
I20260806 02:44:32.434543 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.428930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.28.191.3:35117)
0806 02:44:32.429876 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.429885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.429902 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:44:32.429958 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.429961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.430101 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.430493 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:32.430607 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431292 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.433195 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.433975 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.434075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.434205 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.434316 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.434405 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":130,"threads_started":1}
I20260806 02:44:32.435102 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.429239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.28.191.2:40727)
0806 02:44:32.429650 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.429665 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.429690 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:44:32.429855 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.429861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.430046 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.430183 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:32.430432 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431546 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.432951 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.433810 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.433924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.434088 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.434851 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.434954 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":111,"threads_started":1}
I20260806 02:44:32.434470 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.428870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40727 (local address 127.0.0.1:45048)
0806 02:44:32.429619 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.429632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.429722 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.430195 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.430200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.430223 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:32.430314 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.430320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431710 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.431713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.432823 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.432833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.433934 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.433937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.434012 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.434092 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.434197 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.434316 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":26,"thread_start_us":95,"threads_started":1}
I20260806 02:44:32.435499 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.428219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:45291 (local address 127.0.0.1:40262)
0806 02:44:32.428698 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.428745 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.428861 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.429843 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.429849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.429872 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:32.429990 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.429996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431671 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.431674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.433099 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.433105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.435121 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.435124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.435228 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.435253 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.435300 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.435366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":137,"thread_start_us":63,"threads_started":1}
I20260806 02:44:32.436043 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.428357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.28.191.1:45291)
0806 02:44:32.429328 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.429345 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.429369 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:44:32.429445 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.429451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.429659 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.430167 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:32.430335 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.431531 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.434164 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.434922 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.435667 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.435813 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.435860 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.435932 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":77,"threads_started":1}
I20260806 02:44:32.437572 24951 tablet_service.cc:1467] Processing CreateTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 (DEFAULT_TABLE table=test-table [id=e8a7a794f9e84f02bb02c01237247ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:32.437896 24778 tablet_service.cc:1467] Processing CreateTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 (DEFAULT_TABLE table=test-table [id=e8a7a794f9e84f02bb02c01237247ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:32.438666 24778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c088fa45ea8473ea792a37b4b4fd414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:32.438690 24951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c088fa45ea8473ea792a37b4b4fd414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:32.438688 24604 tablet_service.cc:1467] Processing CreateTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 (DEFAULT_TABLE table=test-table [id=e8a7a794f9e84f02bb02c01237247ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:32.439582 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c088fa45ea8473ea792a37b4b4fd414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:32.445128 25059 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Bootstrap starting.
I20260806 02:44:32.446440 25061 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: Bootstrap starting.
I20260806 02:44:32.446522 25060 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Bootstrap starting.
I20260806 02:44:32.447968 25059 tablet_bootstrap.cc:654] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:32.448138 25060 tablet_bootstrap.cc:654] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:32.448446 25061 tablet_bootstrap.cc:654] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:32.448868 25059 log.cc:824] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:32.449246 25060 log.cc:824] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:32.449599 25061 log.cc:824] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:32.451081 25060 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: No bootstrap required, opened a new log
I20260806 02:44:32.451278 25060 ts_tablet_manager.cc:1397] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:44:32.451439 25061 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: No bootstrap required, opened a new log
I20260806 02:44:32.451439 25059 tablet_bootstrap.cc:492] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: No bootstrap required, opened a new log
I20260806 02:44:32.451614 25061 ts_tablet_manager.cc:1397] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:44:32.451614 25059 ts_tablet_manager.cc:1397] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:44:32.454247 25060 raft_consensus.cc:348] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.454490 25060 raft_consensus.cc:374] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:32.454552 25060 raft_consensus.cc:729] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4172844ca9c24dde9c81a12e5ec61657, State: Initialized, Role: FOLLOWER
I20260806 02:44:32.454968 25060 consensus_queue.cc:260] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.455384 25059 raft_consensus.cc:348] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.455617 25038 heartbeater.cc:503] Master 127.28.191.62:40183 was elected leader, sending a full tablet report...
I20260806 02:44:32.455641 25059 raft_consensus.cc:374] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:32.455687 25059 raft_consensus.cc:729] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaf85c5b0e44294851c3dcfc54ed27e, State: Initialized, Role: FOLLOWER
I20260806 02:44:32.455960 25060 ts_tablet_manager.cc:1428] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260806 02:44:32.456074 25059 consensus_queue.cc:260] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.455384 25061 raft_consensus.cc:348] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.456959 25061 raft_consensus.cc:374] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:32.457020 25059 ts_tablet_manager.cc:1428] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:44:32.457024 25061 raft_consensus.cc:729] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56087ca1fb0f480289fa3c744a3f3522, State: Initialized, Role: FOLLOWER
I20260806 02:44:32.457424 24866 heartbeater.cc:503] Master 127.28.191.62:40183 was elected leader, sending a full tablet report...
I20260806 02:44:32.457455 25061 consensus_queue.cc:260] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.458103 24686 heartbeater.cc:503] Master 127.28.191.62:40183 was elected leader, sending a full tablet report...
I20260806 02:44:32.458436 25061 ts_tablet_manager.cc:1428] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:44:32.475708 25068 raft_consensus.cc:482] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:32.475962 25068 raft_consensus.cc:504] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.477152 25068 leader_election.cc:288] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4172844ca9c24dde9c81a12e5ec61657 (127.28.191.3:35117), acaf85c5b0e44294851c3dcfc54ed27e (127.28.191.2:40727)
I20260806 02:44:32.480902 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.477346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:35117 (local address 127.28.191.1:60393)
0806 02:44:32.477685 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.477697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.477784 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.478031 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.478034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.478053 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:32.478194 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.478202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.478992 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.478996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.479807 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.479817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.480463 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.480470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.480585 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.480611 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.480680 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.480752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":89,"threads_started":1}
I20260806 02:44:32.480942 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.477399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:40727 (local address 127.28.191.1:51791)
0806 02:44:32.477789 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.477804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:32.477908 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:32.478216 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:32.478218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:32.478232 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:32.478370 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.478377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.479105 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.479107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.479844 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:32.479852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.480534 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.480538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:32.480642 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.480677 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:44:32.480758 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:44:32.480819 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260806 02:44:32.480918 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.477460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:60393 (local address 127.28.191.3:35117)
0806 02:44:32.477591 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.477595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.477781 (+ 186us) server_negotiation.cc:407] Connection header received
0806 02:44:32.477874 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.477878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.477924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.478004 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:32.478314 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.478881 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.479940 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.480333 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.480423 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.480488 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.480726 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.480803 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:44:32.480971 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.477497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:51791 (local address 127.28.191.2:40727)
0806 02:44:32.477621 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.477626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.477899 (+ 273us) server_negotiation.cc:407] Connection header received
0806 02:44:32.478038 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.478041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.478111 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.478191 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:32.478477 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.478975 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.479988 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.480404 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.480516 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.480567 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.480789 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.480855 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:32.481560 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c088fa45ea8473ea792a37b4b4fd414" candidate_uuid: "56087ca1fb0f480289fa3c744a3f3522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4172844ca9c24dde9c81a12e5ec61657" is_pre_election: true
I20260806 02:44:32.481575 24803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c088fa45ea8473ea792a37b4b4fd414" candidate_uuid: "56087ca1fb0f480289fa3c744a3f3522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" is_pre_election: true
I20260806 02:44:32.481844 24803 raft_consensus.cc:2454] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56087ca1fb0f480289fa3c744a3f3522 in term 0.
I20260806 02:44:32.481848 24976 raft_consensus.cc:2454] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56087ca1fb0f480289fa3c744a3f3522 in term 0.
I20260806 02:44:32.482307 24546 leader_election.cc:302] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56087ca1fb0f480289fa3c744a3f3522, acaf85c5b0e44294851c3dcfc54ed27e; no voters:
I20260806 02:44:32.482568 25068 raft_consensus.cc:2781] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:32.482653 25068 raft_consensus.cc:482] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:44:32.482695 25068 raft_consensus.cc:3037] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:32.483734 25068 raft_consensus.cc:504] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.484040 25068 leader_election.cc:288] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [CANDIDATE]: Term 1 election: Requested vote from peers 4172844ca9c24dde9c81a12e5ec61657 (127.28.191.3:35117), acaf85c5b0e44294851c3dcfc54ed27e (127.28.191.2:40727)
I20260806 02:44:32.484334 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c088fa45ea8473ea792a37b4b4fd414" candidate_uuid: "56087ca1fb0f480289fa3c744a3f3522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4172844ca9c24dde9c81a12e5ec61657"
I20260806 02:44:32.484364 24803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c088fa45ea8473ea792a37b4b4fd414" candidate_uuid: "56087ca1fb0f480289fa3c744a3f3522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acaf85c5b0e44294851c3dcfc54ed27e"
I20260806 02:44:32.484475 24976 raft_consensus.cc:3037] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:32.484462 24803 raft_consensus.cc:3037] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:32.485613 24803 raft_consensus.cc:2454] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56087ca1fb0f480289fa3c744a3f3522 in term 1.
I20260806 02:44:32.485653 24976 raft_consensus.cc:2454] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56087ca1fb0f480289fa3c744a3f3522 in term 1.
I20260806 02:44:32.485913 24546 leader_election.cc:302] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56087ca1fb0f480289fa3c744a3f3522, acaf85c5b0e44294851c3dcfc54ed27e; no voters:
I20260806 02:44:32.486107 25068 raft_consensus.cc:2781] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:32.486311 25068 raft_consensus.cc:686] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [term 1 LEADER]: Becoming Leader. State: Replica: 56087ca1fb0f480289fa3c744a3f3522, State: Running, Role: LEADER
I20260806 02:44:32.486620 25068 consensus_queue.cc:237] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } }
I20260806 02:44:32.489706 24434 catalog_manager.cc:5495] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56087ca1fb0f480289fa3c744a3f3522 (127.28.191.1). New cstate: current_term: 1 leader_uuid: "56087ca1fb0f480289fa3c744a3f3522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56087ca1fb0f480289fa3c744a3f3522" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 45291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 } health_report { overall_health: UNKNOWN } } }
I20260806 02:44:32.512374 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.508343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.28.191.1:45291)
0806 02:44:32.508544 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.508551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.508583 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:44:32.508680 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.508685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.508769 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.508891 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:32.509236 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.509967 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.510842 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.511058 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.511145 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.511458 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:32.511885 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:32.511977 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.512137 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.512190 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:44:32.518543 24976 raft_consensus.cc:1261] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 1 FOLLOWER]: Refusing update from remote peer 56087ca1fb0f480289fa3c744a3f3522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:32.518543 24803 raft_consensus.cc:1261] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 1 FOLLOWER]: Refusing update from remote peer 56087ca1fb0f480289fa3c744a3f3522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:32.519326 25068 consensus_queue.cc:1035] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4172844ca9c24dde9c81a12e5ec61657" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:32.519552 25076 consensus_queue.cc:1035] T 1c088fa45ea8473ea792a37b4b4fd414 P 56087ca1fb0f480289fa3c744a3f3522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acaf85c5b0e44294851c3dcfc54ed27e" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:32.526958 25080 mvcc.cc:205] Tried to move back new op lower bound from 7315391580228063232 to 7315391580108017664. Current Snapshot: MvccSnapshot[applied={T|T < 7315391580228063232}]
I20260806 02:44:32.528148 25082 mvcc.cc:205] Tried to move back new op lower bound from 7315391580228063232 to 7315391580108017664. Current Snapshot: MvccSnapshot[applied={T|T < 7315391580228063232}]
I20260806 02:44:32.528369 25081 mvcc.cc:205] Tried to move back new op lower bound from 7315391580228063232 to 7315391580108017664. Current Snapshot: MvccSnapshot[applied={T|T < 7315391580228063232}]
I20260806 02:44:32.536475 24434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:32.536664 24434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:32.539273 24434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e810a9920dac4166b87554ef52d49335: Sending DeleteTablet for 3 replicas of tablet 1c088fa45ea8473ea792a37b4b4fd414
I20260806 02:44:32.540180 24951 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:32 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260806 02:44:32.540303 24604 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:32 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:40262
I20260806 02:44:32.540442 24778 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c088fa45ea8473ea792a37b4b4fd414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:32 UTC) from {username='kudu', principal='kudu/127.28.191.62@KRBTEST.COM'} at 127.0.0.1:45048
I20260806 02:44:32.540767 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 24520
I20260806 02:44:32.546762 25095 tablet_replica.cc:330] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: stopping tablet replica
I20260806 02:44:32.547294 25095 raft_consensus.cc:2229] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:32.547809 25095 raft_consensus.cc:2258] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:32.550688 25095 ts_tablet_manager.cc:1905] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:32.554068 25093 tablet_replica.cc:330] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: stopping tablet replica
I20260806 02:44:32.554574 25093 raft_consensus.cc:2229] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:32.554627 25095 ts_tablet_manager.cc:1918] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:32.554733 25095 log.cc:1196] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/1c088fa45ea8473ea792a37b4b4fd414
I20260806 02:44:32.555116 25095 ts_tablet_manager.cc:1939] T 1c088fa45ea8473ea792a37b4b4fd414 P acaf85c5b0e44294851c3dcfc54ed27e: Deleting consensus metadata
I20260806 02:44:32.555169 25093 raft_consensus.cc:2258] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:32.556663 24417 catalog_manager.cc:4841] TS acaf85c5b0e44294851c3dcfc54ed27e (127.28.191.2:40727): tablet 1c088fa45ea8473ea792a37b4b4fd414 (table test-table [id=e8a7a794f9e84f02bb02c01237247ee5]) successfully deleted
I20260806 02:44:32.556955 25093 ts_tablet_manager.cc:1905] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:32.559229 25093 ts_tablet_manager.cc:1918] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:32.559316 25093 log.cc:1196] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/1c088fa45ea8473ea792a37b4b4fd414
I20260806 02:44:32.559654 25093 ts_tablet_manager.cc:1939] T 1c088fa45ea8473ea792a37b4b4fd414 P 4172844ca9c24dde9c81a12e5ec61657: Deleting consensus metadata
I20260806 02:44:32.560825 24415 catalog_manager.cc:4841] TS 4172844ca9c24dde9c81a12e5ec61657 (127.28.191.3:35117): tablet 1c088fa45ea8473ea792a37b4b4fd414 (table test-table [id=e8a7a794f9e84f02bb02c01237247ee5]) successfully deleted
W20260806 02:44:32.561792 24417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:45291 (error 108)
I20260806 02:44:32.562016 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 24693
I20260806 02:44:32.565634 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.565244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:45291 (local address 127.0.0.1:40282)
0806 02:44:32.565447 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:32.565533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:45291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 02:44:32.565798 24417 catalog_manager.cc:4584] TS 56087ca1fb0f480289fa3c744a3f3522 (127.28.191.1:45291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c088fa45ea8473ea792a37b4b4fd414: Network error: Client connection negotiation failed: client connection to 127.28.191.1:45291: connect: Connection refused (error 111)
I20260806 02:44:32.572723 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 24873
I20260806 02:44:32.581914 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 24394
2026-08-06T02:44:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2492 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:32Z Disabled control of system clock
I20260806 02:44:32.616098 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:44657
--webserver_interface=127.28.191.62
--webserver_port=0
--builtin_ntp_servers=127.28.191.20:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:32.730436 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:32.730679 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:32.730713 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:32.734613 25103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:32.734664 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:32.734680 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:32.734696 25103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:32.734711 25103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:32.739352 25103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:44657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:32.740293 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:32.741643 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:32.750056 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:32.750196 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:32.750196 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:32.750418 25103 server_base.cc:1034] running on GCE node
I20260806 02:44:32.750980 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:32.751971 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:32.753170 25103 hybrid_clock.cc:648] HybridClock initialized: now 1785984272753145 us; error 37 us; skew 500 ppm
I20260806 02:44:32.759147 25103 webserver.cc:460] Webserver started at http://127.28.191.62:42897/ using document root <none> and password file <none>
I20260806 02:44:32.759738 25103 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:32.759835 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:32.760083 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:32.762221 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "c786274617ec4226a0959ff1f1a67ade"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "1084fe99121eeb39c5e92dcc63fea7dd"
server_key_iv: "24749ac0c64c8a6c160020fe78a62a06"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.762796 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "c786274617ec4226a0959ff1f1a67ade"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "1084fe99121eeb39c5e92dcc63fea7dd"
server_key_iv: "24749ac0c64c8a6c160020fe78a62a06"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.766419 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:32.768887 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:32.770179 25103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:32.770318 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "c786274617ec4226a0959ff1f1a67ade"
format_stamp: "Formatted at 2026-08-06 02:44:32 on dist-test-slave-tfpf"
server_key: "1084fe99121eeb39c5e92dcc63fea7dd"
server_key_iv: "24749ac0c64c8a6c160020fe78a62a06"
server_key_version: "encryptionkey@0"
I20260806 02:44:32.770427 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:32.785298 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:32.785945 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:32.786155 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:32.792922 25103 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:44657
I20260806 02:44:32.792932 25190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:44657 every 8 connection(s)
I20260806 02:44:32.793898 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:32.796989 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:32.802284 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25103
I20260806 02:44:32.802429 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:32.802434 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: Bootstrap starting.
I20260806 02:44:32.802774 29436 external_mini_cluster.cc:1428] Setting key 3aaed4b33834c113efc307e649d48df7
I20260806 02:44:32.805209 25194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:32.805938 25194 log.cc:824] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:32.807883 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: No bootstrap required, opened a new log
I20260806 02:44:32.811667 25194 raft_consensus.cc:348] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } }
I20260806 02:44:32.811874 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:32.804156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.28.191.62:44657)
0806 02:44:32.804782 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:32.804795 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:32.804871 (+ 76us) server_negotiation.cc:407] Connection header received
0806 02:44:32.805578 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:32.805596 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:32.806014 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:32.806353 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:32.806787 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.807715 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.808655 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:32.809190 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:32.809348 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:32.810230 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:32.810235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:32.810247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:32.810613 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:32.810665 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:32.810684 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:32.810788 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:32.810946 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 02:44:32.811194 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260806 02:44:32.811904 25194 raft_consensus.cc:374] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:32.811987 25194 raft_consensus.cc:729] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c786274617ec4226a0959ff1f1a67ade, State: Initialized, Role: FOLLOWER
I20260806 02:44:32.812469 25194 consensus_queue.cc:260] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } }
I20260806 02:44:32.812626 25194 raft_consensus.cc:388] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:32.812678 25194 raft_consensus.cc:482] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:32.812760 25194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:32.813700 25194 raft_consensus.cc:504] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } }
I20260806 02:44:32.814093 25194 leader_election.cc:302] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c786274617ec4226a0959ff1f1a67ade; no voters:
I20260806 02:44:32.814435 25194 leader_election.cc:288] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:32.814564 25201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:32.814813 25201 raft_consensus.cc:686] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [term 1 LEADER]: Becoming Leader. State: Replica: c786274617ec4226a0959ff1f1a67ade, State: Running, Role: LEADER
I20260806 02:44:32.815151 25201 consensus_queue.cc:237] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } }
I20260806 02:44:32.815676 25194 sys_catalog.cc:564] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:32.816797 25202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c786274617ec4226a0959ff1f1a67ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } } }
I20260806 02:44:32.816924 25202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:32.817328 25212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:32.817703 25203 sys_catalog.cc:455] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader c786274617ec4226a0959ff1f1a67ade. Latest consensus state: current_term: 1 leader_uuid: "c786274617ec4226a0959ff1f1a67ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c786274617ec4226a0959ff1f1a67ade" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44657 } } }
I20260806 02:44:32.817826 25203 sys_catalog.cc:458] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:32.820093 25212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:32.825677 25212 catalog_manager.cc:1347] Generated new cluster ID: 1b369e0dc66b4ff6b69d9b8da0e8a44e
I20260806 02:44:32.825752 25212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:32.842459 25212 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:32.843528 25212 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:32.848340 25212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: Generated new TSK 0
I20260806 02:44:32.848984 25212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:32.854683 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.1:0
--local_ip_for_outbound_sockets=127.28.191.1
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--builtin_ntp_servers=127.28.191.20:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:44:32.991230 25224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:32.991489 25224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:32.991564 25224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:32.995813 25224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:32.995882 25224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:32.995965 25224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.1
I20260806 02:44:33.000989 25224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.191.1
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.1
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:33.002251 25224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:33.003743 25224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:33.013762 25235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:33.013785 25231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.013911 25224 server_base.cc:1034] running on GCE node
W20260806 02:44:33.013772 25232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.014432 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:33.015075 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:33.016283 25224 hybrid_clock.cc:648] HybridClock initialized: now 1785984273016262 us; error 32 us; skew 500 ppm
I20260806 02:44:33.018332 25224 webserver.cc:460] Webserver started at http://127.28.191.1:35295/ using document root <none> and password file <none>
I20260806 02:44:33.018932 25224 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:33.019026 25224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:33.019270 25224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:33.021397 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data/instance:
uuid: "9014a94d02704d63b5d9228a5e25c8dc"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "57d49e9fbab655732f696181577ca326"
server_key_iv: "c14032ce0ae6cd13213adf5c42740b76"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.021974 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance:
uuid: "9014a94d02704d63b5d9228a5e25c8dc"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "57d49e9fbab655732f696181577ca326"
server_key_iv: "c14032ce0ae6cd13213adf5c42740b76"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.025588 25224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:44:33.027895 25242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.029136 25224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:33.029261 25224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal
uuid: "9014a94d02704d63b5d9228a5e25c8dc"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "57d49e9fbab655732f696181577ca326"
server_key_iv: "c14032ce0ae6cd13213adf5c42740b76"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.029369 25224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:33.039440 25224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:33.040033 25224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:33.040220 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:33.040771 25224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:33.041783 25224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:33.041872 25224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.041941 25224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:33.041992 25224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.051018 25224 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.1:36609
I20260806 02:44:33.051056 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.1:36609 every 8 connection(s)
I20260806 02:44:33.051857 25224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/data/info.pb
I20260806 02:44:33.053536 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25224
I20260806 02:44:33.053676 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-0/wal/instance
I20260806 02:44:33.054023 29436 external_mini_cluster.cc:1428] Setting key 7dfeb4b5909c7f5905434bab7d56890c
I20260806 02:44:33.056383 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.2:0
--local_ip_for_outbound_sockets=127.28.191.2
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--builtin_ntp_servers=127.28.191.20:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:33.062886 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.053872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.1:43093 (local address 127.28.191.62:44657)
0806 02:44:33.054015 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.054019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.055310 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 02:44:33.057236 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.057238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.057282 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.057350 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:33.058517 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.059383 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.060521 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.060939 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.061086 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.062292 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:33.062299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:33.062305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:33.062323 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:33.062350 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:33.062363 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:33.062452 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.062596 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.062669 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:44:33.063619 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.054341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44657 (local address 127.28.191.1:43093)
0806 02:44:33.055150 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.055205 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.056994 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.057517 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.057528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.057640 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:33.058329 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.058353 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.059523 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.059526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.060386 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.060395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.061139 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.061143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.061294 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.062028 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:33.062030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:33.062072 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:33.062134 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:33.062451 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:33.062458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:33.062462 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.062562 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.062769 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":171,"threads_started":1}
I20260806 02:44:33.064823 25387 heartbeater.cc:347] Connected to a master server at 127.28.191.62:44657
I20260806 02:44:33.065129 25387 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:33.065656 25387 heartbeater.cc:511] Master 127.28.191.62:44657 requested a full tablet report, sending...
I20260806 02:44:33.067132 25143 ts_manager.cc:194] Registered new tserver with Master: 9014a94d02704d63b5d9228a5e25c8dc (127.28.191.1:36609)
I20260806 02:44:33.068254 25143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.1:43093
W20260806 02:44:33.169943 25391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:33.170208 25391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:33.170276 25391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:33.174352 25391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:33.174414 25391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:33.174499 25391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.2
I20260806 02:44:33.179112 25391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.191.2
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.2
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:33.180099 25391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:33.181460 25391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:33.190702 25403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:33.190711 25398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.190851 25391 server_base.cc:1034] running on GCE node
W20260806 02:44:33.190742 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.191344 25391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:33.191931 25391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:33.193130 25391 hybrid_clock.cc:648] HybridClock initialized: now 1785984273193102 us; error 40 us; skew 500 ppm
I20260806 02:44:33.194759 25391 webserver.cc:460] Webserver started at http://127.28.191.2:43007/ using document root <none> and password file <none>
I20260806 02:44:33.195272 25391 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:33.195330 25391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:33.195485 25391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:33.197520 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data/instance:
uuid: "4eac4ab291be49e68e17010b2e4331b6"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "2e5cebb8e1ee99378ba26e96e0b0373b"
server_key_iv: "dcfc2831e8acef41510fcfad4cc578a9"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.198017 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance:
uuid: "4eac4ab291be49e68e17010b2e4331b6"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "2e5cebb8e1ee99378ba26e96e0b0373b"
server_key_iv: "dcfc2831e8acef41510fcfad4cc578a9"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.201478 25391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:44:33.203769 25410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.204903 25391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:33.204993 25391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal
uuid: "4eac4ab291be49e68e17010b2e4331b6"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "2e5cebb8e1ee99378ba26e96e0b0373b"
server_key_iv: "dcfc2831e8acef41510fcfad4cc578a9"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.205194 25391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:33.222834 25391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:33.223451 25391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:33.223640 25391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:33.224185 25391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:33.225100 25391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:33.225160 25391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.225221 25391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:33.225260 25391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.233788 25391 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.2:33853
I20260806 02:44:33.233829 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.2:33853 every 8 connection(s)
I20260806 02:44:33.234622 25391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/data/info.pb
I20260806 02:44:33.243588 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.236306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:44013 (local address 127.28.191.62:44657)
0806 02:44:33.236441 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.236444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.237476 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 02:44:33.238872 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.238875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.238932 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.239038 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:33.239883 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.240413 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.241658 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.241878 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.241990 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.243110 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:33.243113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:33.243115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:33.243127 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:33.243144 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:33.243152 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:33.243228 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.243390 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.243449 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:44:33.244280 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25391
I20260806 02:44:33.244366 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.236668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44657 (local address 127.28.191.2:44013)
0806 02:44:33.237339 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.237378 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.238667 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.239205 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.239211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.239289 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:33.239741 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.239753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.240554 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.240558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.241507 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.241517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.242028 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.242033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.242197 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.242844 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:33.242846 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:33.242883 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:33.242942 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:33.243230 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:33.243242 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:33.243250 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.243378 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.243643 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260806 02:44:33.244438 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal/instance
I20260806 02:44:33.244787 29436 external_mini_cluster.cc:1428] Setting key 0476c192cbc4b31da18844bcca9a1d11
I20260806 02:44:33.245618 25562 heartbeater.cc:347] Connected to a master server at 127.28.191.62:44657
I20260806 02:44:33.245844 25562 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:33.246450 25562 heartbeater.cc:511] Master 127.28.191.62:44657 requested a full tablet report, sending...
I20260806 02:44:33.246898 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.191.3:0
--local_ip_for_outbound_sockets=127.28.191.3
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--builtin_ntp_servers=127.28.191.20:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:44:33.247491 25143 ts_manager.cc:194] Registered new tserver with Master: 4eac4ab291be49e68e17010b2e4331b6 (127.28.191.2:33853)
I20260806 02:44:33.248028 25143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.2:44013
W20260806 02:44:33.363260 25565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:33.363495 25565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:33.363533 25565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:33.367518 25565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:33.367568 25565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:33.367650 25565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.191.3
I20260806 02:44:33.372344 25565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.191.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.191.3
--webserver_port=0
--tserver_master_addrs=127.28.191.62:44657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.191.3
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:33.373468 25565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:33.374817 25565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:33.384562 25572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:33.384577 25571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.384660 25565 server_base.cc:1034] running on GCE node
W20260806 02:44:33.384564 25574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:33.385265 25565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:33.385823 25565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:33.386981 25565 hybrid_clock.cc:648] HybridClock initialized: now 1785984273386961 us; error 48 us; skew 500 ppm
I20260806 02:44:33.388778 25565 webserver.cc:460] Webserver started at http://127.28.191.3:35809/ using document root <none> and password file <none>
I20260806 02:44:33.389523 25565 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:33.389635 25565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:33.389865 25565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:33.391968 25565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data/instance:
uuid: "08ead751553e4810b8d6f429bdba8936"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "fcd1001bac34956c5ced5333dad8f71b"
server_key_iv: "30551a73f45f5970abefb254d7b2d74b"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.392590 25565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance:
uuid: "08ead751553e4810b8d6f429bdba8936"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "fcd1001bac34956c5ced5333dad8f71b"
server_key_iv: "30551a73f45f5970abefb254d7b2d74b"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.396770 25565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:44:33.399250 25585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.400353 25565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:33.400506 25565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal
uuid: "08ead751553e4810b8d6f429bdba8936"
format_stamp: "Formatted at 2026-08-06 02:44:33 on dist-test-slave-tfpf"
server_key: "fcd1001bac34956c5ced5333dad8f71b"
server_key_iv: "30551a73f45f5970abefb254d7b2d74b"
server_key_version: "encryptionkey@0"
I20260806 02:44:33.400626 25565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:33.428638 25565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:33.429445 25565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:33.429677 25565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:33.430310 25565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:44:33.431339 25565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:44:33.431418 25565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.431506 25565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:44:33.431563 25565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:33.441080 25565 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.3:36875
I20260806 02:44:33.441159 25728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.3:36875 every 8 connection(s)
I20260806 02:44:33.442044 25565 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/data/info.pb
I20260806 02:44:33.444377 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25565
I20260806 02:44:33.444486 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal/instance
I20260806 02:44:33.444828 29436 external_mini_cluster.cc:1428] Setting key d6fb2a31861ebf4676c77919f0f2dd31
I20260806 02:44:33.451674 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.443960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.3:46591 (local address 127.28.191.62:44657)
0806 02:44:33.444192 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.444199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.445287 (+ 1088us) server_negotiation.cc:407] Connection header received
0806 02:44:33.446967 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.446970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.447049 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.447144 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:33.448069 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.448672 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.449841 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.450100 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.450188 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.451249 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:33.451252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:33.451254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:33.451266 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:33.451284 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:33.451293 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:33.451356 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.451515 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.451569 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:44:33.452445 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.444344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44657 (local address 127.28.191.3:46591)
0806 02:44:33.445120 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.445172 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.446749 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.447369 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.447384 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.447467 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:44:33.447938 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.447950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.448782 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.448785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.449725 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.449736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.450231 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.450233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.450373 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.451007 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:44:33.451009 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:44:33.451053 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:44:33.451112 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:44:33.451371 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:44:33.451377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:44:33.451381 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.451471 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.451671 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":162,"threads_started":1}
I20260806 02:44:33.453445 25729 heartbeater.cc:347] Connected to a master server at 127.28.191.62:44657
I20260806 02:44:33.453631 25729 heartbeater.cc:464] Registering TS with master...
I20260806 02:44:33.454097 25729 heartbeater.cc:511] Master 127.28.191.62:44657 requested a full tablet report, sending...
I20260806 02:44:33.454984 25143 ts_manager.cc:194] Registered new tserver with Master: 08ead751553e4810b8d6f429bdba8936 (127.28.191.3:36875)
I20260806 02:44:33.455569 25143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.191.3:46591
I20260806 02:44:33.458827 29436 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:44:33.464143 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.460666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.28.191.62:44657)
0806 02:44:33.460803 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.460808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.461010 (+ 202us) server_negotiation.cc:407] Connection header received
0806 02:44:33.461178 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.461180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.461218 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.461281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:33.461509 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.461976 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.462740 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.462938 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.463010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.463699 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:33.463702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:33.463704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:33.463716 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:33.463733 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:33.463740 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:33.463797 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.463895 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.463942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:44:33.466850 25143 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:44:33.469928 25143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:44:33.484324 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.28.191.3:36875)
0806 02:44:33.479566 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.479578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.479600 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:44:33.479675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.479680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.479890 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.480096 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:33.480398 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.481338 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.482914 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.483657 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.483782 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.483971 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.484072 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.484162 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":162,"threads_started":1}
I20260806 02:44:33.484319 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:36875 (local address 127.0.0.1:44490)
0806 02:44:33.478848 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.478897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.479028 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.480082 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.480085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.480103 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:33.480255 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.480263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.481439 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.481443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.482786 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.482796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.483784 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.483788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.483894 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.483931 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.484032 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.484162 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":53,"thread_start_us":80,"threads_started":1}
I20260806 02:44:33.484889 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:36609 (local address 127.0.0.1:41974)
0806 02:44:33.480257 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.480270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.480422 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.480905 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.480908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.480928 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:33.481004 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.481010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.482163 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.482167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.483305 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.483316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.484520 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.484523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.484621 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.484655 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.484718 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.484765 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":101,"threads_started":1}
I20260806 02:44:33.484889 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.2:33853 (local address 127.0.0.1:41220)
0806 02:44:33.478925 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.478940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.479032 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.480016 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.480024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.480051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:33.480238 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.480252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.481622 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.481625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.483081 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.483089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.484268 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.484273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.484356 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.484377 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.484658 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.484742 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":407,"thread_start_us":87,"threads_started":1}
I20260806 02:44:33.485301 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.28.191.2:33853)
0806 02:44:33.479529 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.479538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.479561 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:44:33.479623 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.479627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.479832 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.480008 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:33.480366 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.481515 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.483198 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.483909 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.484014 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.484195 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.484955 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.485099 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":142,"threads_started":1}
I20260806 02:44:33.487238 25645 tablet_service.cc:1467] Processing CreateTablet for tablet eca1171499254ee58a0d6a0a80cce9aa (DEFAULT_TABLE table=test-table [id=4ec059ee52e34e68af110513242185b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:33.487587 25476 tablet_service.cc:1467] Processing CreateTablet for tablet eca1171499254ee58a0d6a0a80cce9aa (DEFAULT_TABLE table=test-table [id=4ec059ee52e34e68af110513242185b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:33.488253 25645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca1171499254ee58a0d6a0a80cce9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:33.488505 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca1171499254ee58a0d6a0a80cce9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:33.493535 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.478929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.28.191.1:36609)
0806 02:44:33.479378 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.479386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.480354 (+ 968us) server_negotiation.cc:407] Connection header received
0806 02:44:33.480584 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.480590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.480760 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.480901 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:33.481150 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.482046 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.483433 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.484197 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.484302 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.484465 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.485190 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.493333 (+ 8143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":105,"threads_started":1}
I20260806 02:44:33.494786 25752 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Bootstrap starting.
I20260806 02:44:33.495605 25753 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Bootstrap starting.
I20260806 02:44:33.496361 25305 tablet_service.cc:1467] Processing CreateTablet for tablet eca1171499254ee58a0d6a0a80cce9aa (DEFAULT_TABLE table=test-table [id=4ec059ee52e34e68af110513242185b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:44:33.497015 25752 tablet_bootstrap.cc:654] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:33.497524 25305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca1171499254ee58a0d6a0a80cce9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:33.497870 25753 tablet_bootstrap.cc:654] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:33.497888 25752 log.cc:824] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:33.498709 25753 log.cc:824] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:33.499786 25752 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: No bootstrap required, opened a new log
I20260806 02:44:33.499989 25752 ts_tablet_manager.cc:1397] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:44:33.500772 25753 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: No bootstrap required, opened a new log
I20260806 02:44:33.501055 25753 ts_tablet_manager.cc:1397] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:44:33.503298 25752 raft_consensus.cc:348] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.503535 25752 raft_consensus.cc:374] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:33.503620 25752 raft_consensus.cc:729] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ead751553e4810b8d6f429bdba8936, State: Initialized, Role: FOLLOWER
I20260806 02:44:33.504140 25752 consensus_queue.cc:260] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.504797 25729 heartbeater.cc:503] Master 127.28.191.62:44657 was elected leader, sending a full tablet report...
I20260806 02:44:33.505438 25761 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Bootstrap starting.
I20260806 02:44:33.505798 25753 raft_consensus.cc:348] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.506155 25753 raft_consensus.cc:374] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:33.506259 25753 raft_consensus.cc:729] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eac4ab291be49e68e17010b2e4331b6, State: Initialized, Role: FOLLOWER
I20260806 02:44:33.506425 25752 ts_tablet_manager.cc:1428] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:44:33.506780 25753 consensus_queue.cc:260] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.507459 25562 heartbeater.cc:503] Master 127.28.191.62:44657 was elected leader, sending a full tablet report...
I20260806 02:44:33.508052 25761 tablet_bootstrap.cc:654] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:33.508324 25753 ts_tablet_manager.cc:1428] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:44:33.508994 25761 log.cc:824] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:33.510882 25761 tablet_bootstrap.cc:492] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: No bootstrap required, opened a new log
I20260806 02:44:33.511075 25761 ts_tablet_manager.cc:1397] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:44:33.513993 25762 raft_consensus.cc:482] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:44:33.514146 25762 raft_consensus.cc:504] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.514302 25761 raft_consensus.cc:348] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.514530 25761 raft_consensus.cc:374] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:33.514611 25761 raft_consensus.cc:729] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9014a94d02704d63b5d9228a5e25c8dc, State: Initialized, Role: FOLLOWER
I20260806 02:44:33.514987 25761 consensus_queue.cc:260] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.515650 25387 heartbeater.cc:503] Master 127.28.191.62:44657 was elected leader, sending a full tablet report...
I20260806 02:44:33.515904 25762 leader_election.cc:288] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ead751553e4810b8d6f429bdba8936 (127.28.191.3:36875), 9014a94d02704d63b5d9228a5e25c8dc (127.28.191.1:36609)
I20260806 02:44:33.518814 25761 ts_tablet_manager.cc:1428] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:44:33.521230 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.515500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.3:36875 (local address 127.28.191.2:57161)
0806 02:44:33.517297 (+ 1797us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.517310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.517408 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.517788 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.517791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.517810 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:33.517919 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.517925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.518925 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.518929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.519885 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.519898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.520723 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.520727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.520832 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.520864 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.520979 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.521102 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724}
I20260806 02:44:33.521230 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.515760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:57161 (local address 127.28.191.3:36875)
0806 02:44:33.517604 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.517608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.517618 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:44:33.517656 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.517658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.517691 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.517749 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:33.518081 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.518794 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.520056 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.520593 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.520734 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.520791 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.521009 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.521106 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1745}
I20260806 02:44:33.521862 25669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca1171499254ee58a0d6a0a80cce9aa" candidate_uuid: "4eac4ab291be49e68e17010b2e4331b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ead751553e4810b8d6f429bdba8936" is_pre_election: true
I20260806 02:44:33.522158 25669 raft_consensus.cc:2454] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eac4ab291be49e68e17010b2e4331b6 in term 0.
I20260806 02:44:33.522660 25412 leader_election.cc:302] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ead751553e4810b8d6f429bdba8936, 4eac4ab291be49e68e17010b2e4331b6; no voters:
I20260806 02:44:33.522936 25762 raft_consensus.cc:2781] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:44:33.523047 25762 raft_consensus.cc:482] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:44:33.523102 25762 raft_consensus.cc:3037] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:33.523582 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.515580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:36609 (local address 127.28.191.2:42461)
0806 02:44:33.519195 (+ 3615us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.519206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:44:33.519321 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:44:33.519573 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:44:33.519576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:44:33.519591 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:44:33.519717 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.519722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.520461 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.520464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.521524 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:44:33.521535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.523273 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.523276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:44:33.523348 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.523368 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:44:33.523428 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 02:44:33.523473 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3560,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260806 02:44:33.523648 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.515937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.191.2:42461 (local address 127.28.191.1:36609)
0806 02:44:33.517878 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.517881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.519287 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 02:44:33.519426 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.519429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.519470 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.519560 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:44:33.519826 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.520320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.522499 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.523112 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.523270 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.523340 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.523450 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.523532 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1876}
I20260806 02:44:33.524169 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca1171499254ee58a0d6a0a80cce9aa" candidate_uuid: "4eac4ab291be49e68e17010b2e4331b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9014a94d02704d63b5d9228a5e25c8dc" is_pre_election: true
I20260806 02:44:33.524398 25331 raft_consensus.cc:2454] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eac4ab291be49e68e17010b2e4331b6 in term 0.
I20260806 02:44:33.524708 25762 raft_consensus.cc:504] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.525203 25762 leader_election.cc:288] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [CANDIDATE]: Term 1 election: Requested vote from peers 08ead751553e4810b8d6f429bdba8936 (127.28.191.3:36875), 9014a94d02704d63b5d9228a5e25c8dc (127.28.191.1:36609)
I20260806 02:44:33.525496 25669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca1171499254ee58a0d6a0a80cce9aa" candidate_uuid: "4eac4ab291be49e68e17010b2e4331b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ead751553e4810b8d6f429bdba8936"
I20260806 02:44:33.525552 25331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca1171499254ee58a0d6a0a80cce9aa" candidate_uuid: "4eac4ab291be49e68e17010b2e4331b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9014a94d02704d63b5d9228a5e25c8dc"
I20260806 02:44:33.525652 25331 raft_consensus.cc:3037] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:33.525637 25669 raft_consensus.cc:3037] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:33.526925 25331 raft_consensus.cc:2454] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eac4ab291be49e68e17010b2e4331b6 in term 1.
I20260806 02:44:33.526925 25669 raft_consensus.cc:2454] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eac4ab291be49e68e17010b2e4331b6 in term 1.
I20260806 02:44:33.527235 25417 leader_election.cc:302] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eac4ab291be49e68e17010b2e4331b6, 9014a94d02704d63b5d9228a5e25c8dc; no voters:
I20260806 02:44:33.527411 25762 raft_consensus.cc:2781] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:33.527647 25762 raft_consensus.cc:686] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 1 LEADER]: Becoming Leader. State: Replica: 4eac4ab291be49e68e17010b2e4331b6, State: Running, Role: LEADER
I20260806 02:44:33.528129 25762 consensus_queue.cc:237] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } }
I20260806 02:44:33.531193 25143 catalog_manager.cc:5495] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eac4ab291be49e68e17010b2e4331b6 (127.28.191.2). New cstate: current_term: 1 leader_uuid: "4eac4ab291be49e68e17010b2e4331b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eac4ab291be49e68e17010b2e4331b6" member_type: VOTER last_known_addr { host: "127.28.191.2" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20260806 02:44:33.546936 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.543711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.28.191.2:33853)
0806 02:44:33.543833 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:33.543838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:33.543860 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:44:33.543993 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:33.543995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:33.544046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:33.544134 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:44:33.544325 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.544796 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.545681 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:33.545892 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:33.545981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:33.546262 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:44:33.546653 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:44:33.546744 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:33.546796 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:44:33.546840 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:33.552434 25669 raft_consensus.cc:1261] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 1 FOLLOWER]: Refusing update from remote peer 4eac4ab291be49e68e17010b2e4331b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:44:33.552436 25331 raft_consensus.cc:1261] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc [term 1 FOLLOWER]: Refusing update from remote peer 4eac4ab291be49e68e17010b2e4331b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 02:44:33.552999 25388 tablet.cc:2369] T eca1171499254ee58a0d6a0a80cce9aa P 9014a94d02704d63b5d9228a5e25c8dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:44:33.553217 25769 consensus_queue.cc:1035] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ead751553e4810b8d6f429bdba8936" member_type: VOTER last_known_addr { host: "127.28.191.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:33.553474 25762 consensus_queue.cc:1035] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9014a94d02704d63b5d9228a5e25c8dc" member_type: VOTER last_known_addr { host: "127.28.191.1" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:44:33.561715 25774 mvcc.cc:205] Tried to move back new op lower bound from 7315391584464035840 to 7315391584373825536. Current Snapshot: MvccSnapshot[applied={T|T < 7315391584464035840}]
I20260806 02:44:33.562584 25773 mvcc.cc:205] Tried to move back new op lower bound from 7315391584464035840 to 7315391584373825536. Current Snapshot: MvccSnapshot[applied={T|T < 7315391584464035840}]
I20260806 02:44:33.570377 25143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:33.570573 25143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:44:33.572979 25143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c786274617ec4226a0959ff1f1a67ade: Sending DeleteTablet for 3 replicas of tablet eca1171499254ee58a0d6a0a80cce9aa
I20260806 02:44:33.573750 25305 tablet_service.cc:1514] Processing DeleteTablet for tablet eca1171499254ee58a0d6a0a80cce9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:33 UTC) from {username='slave'} at 127.0.0.1:41974
I20260806 02:44:33.573796 25645 tablet_service.cc:1514] Processing DeleteTablet for tablet eca1171499254ee58a0d6a0a80cce9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:33 UTC) from {username='slave'} at 127.0.0.1:44490
I20260806 02:44:33.573848 25476 tablet_service.cc:1514] Processing DeleteTablet for tablet eca1171499254ee58a0d6a0a80cce9aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:44:33 UTC) from {username='slave'} at 127.0.0.1:41220
I20260806 02:44:33.574333 25783 tablet_replica.cc:330] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: stopping tablet replica
I20260806 02:44:33.574362 25782 tablet_replica.cc:330] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: stopping tablet replica
I20260806 02:44:33.574680 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25224
I20260806 02:44:33.574685 25782 raft_consensus.cc:2229] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:44:33.574988 25782 raft_consensus.cc:2258] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:33.578365 25783 raft_consensus.cc:2229] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:44:33.578853 25783 raft_consensus.cc:2258] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:44:33.580588 25783 ts_tablet_manager.cc:1905] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:33.583793 25783 ts_tablet_manager.cc:1918] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:33.583907 25783 log.cc:1196] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-1/wal/wals/eca1171499254ee58a0d6a0a80cce9aa
I20260806 02:44:33.584266 25783 ts_tablet_manager.cc:1939] T eca1171499254ee58a0d6a0a80cce9aa P 4eac4ab291be49e68e17010b2e4331b6: Deleting consensus metadata
W20260806 02:44:33.585163 25126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.191.1:36609 (error 108)
I20260806 02:44:33.585405 25782 ts_tablet_manager.cc:1905] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:44:33.585640 25124 catalog_manager.cc:4841] TS 4eac4ab291be49e68e17010b2e4331b6 (127.28.191.2:33853): tablet eca1171499254ee58a0d6a0a80cce9aa (table test-table [id=4ec059ee52e34e68af110513242185b3]) successfully deleted
I20260806 02:44:33.586110 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25391
I20260806 02:44:33.591390 25782 ts_tablet_manager.cc:1918] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:44:33.591480 25782 log.cc:1196] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Deleting WAL directory at /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984016215184-29436-0/minicluster-data/ts-2/wal/wals/eca1171499254ee58a0d6a0a80cce9aa
I20260806 02:44:33.591739 25782 ts_tablet_manager.cc:1939] T eca1171499254ee58a0d6a0a80cce9aa P 08ead751553e4810b8d6f429bdba8936: Deleting consensus metadata
I20260806 02:44:33.593135 25121 catalog_manager.cc:4841] TS 08ead751553e4810b8d6f429bdba8936 (127.28.191.3:36875): tablet eca1171499254ee58a0d6a0a80cce9aa (table test-table [id=4ec059ee52e34e68af110513242185b3]) successfully deleted
I20260806 02:44:33.594216 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:33.593865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.1:36609 (local address 127.0.0.1:41978)
0806 02:44:33.594023 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:44:33.594112 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.1:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:44:33.594419 25126 catalog_manager.cc:4584] TS 9014a94d02704d63b5d9228a5e25c8dc (127.28.191.1:36609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eca1171499254ee58a0d6a0a80cce9aa: Network error: Client connection negotiation failed: client connection to 127.28.191.1:36609: connect: Connection refused (error 111)
I20260806 02:44:33.599332 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25565
I20260806 02:44:33.611256 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25103
2026-08-06T02:44:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1032 ms)
[----------] 6 tests from EncryptionPolicyTest (11270 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:33 dist-test-slave-tfpf krb5kdc[25788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:33 dist-test-slave-tfpf krb5kdc[25788](info): set up 2 sockets
Aug 06 02:44:33 dist-test-slave-tfpf krb5kdc[25788](info): commencing operation
krb5kdc: starting...
W20260806 02:44:34.827198 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:34 dist-test-slave-tfpf krb5kdc[25788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:34.953358 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:35.078351 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:35.078661 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:35.078742 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:35.082767 25804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:35.082823 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:35.082842 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:35.082858 25804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:35.082909 25804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:35.088022 25804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:35.089129 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:35.090132 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:35.098578 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:35.098608 25815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:35.098695 25804 server_base.cc:1034] running on GCE node
W20260806 02:44:35.098623 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:35.099524 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:35.100696 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:35.101897 25804 hybrid_clock.cc:648] HybridClock initialized: now 1785984275101876 us; error 33 us; skew 500 ppm
Aug 06 02:44:35 dist-test-slave-tfpf krb5kdc[25788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:35.104604 25804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:35.105542 25804 webserver.cc:460] Webserver started at http://127.0.0.1:42117/ using document root <none> and password file <none>
I20260806 02:44:35.106063 25804 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:35.106122 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:35.106288 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:35.108294 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "f839946eac7b45d58be4b9083fbb672f"
format_stamp: "Formatted at 2026-08-06 02:44:35 on dist-test-slave-tfpf"
server_key: "8b7d53581fbfc80ffa90511edc59010f"
server_key_iv: "9b46a62a335917e5f569db995144ff96"
server_key_version: "encryptionkey@0"
I20260806 02:44:35.108808 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "f839946eac7b45d58be4b9083fbb672f"
format_stamp: "Formatted at 2026-08-06 02:44:35 on dist-test-slave-tfpf"
server_key: "8b7d53581fbfc80ffa90511edc59010f"
server_key_iv: "9b46a62a335917e5f569db995144ff96"
server_key_version: "encryptionkey@0"
I20260806 02:44:35.112717 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:44:35.115234 25824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:35.116736 25804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:44:35.116882 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "f839946eac7b45d58be4b9083fbb672f"
format_stamp: "Formatted at 2026-08-06 02:44:35 on dist-test-slave-tfpf"
server_key: "8b7d53581fbfc80ffa90511edc59010f"
server_key_iv: "9b46a62a335917e5f569db995144ff96"
server_key_version: "encryptionkey@0"
I20260806 02:44:35.116995 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:35.131203 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:35.134536 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:35.134757 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:35.142072 25804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45587
I20260806 02:44:35.142167 25902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45587 every 8 connection(s)
I20260806 02:44:35.143071 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:35.145941 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:35.149976 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25804
I20260806 02:44:35.150094 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:35.150409 29436 external_mini_cluster.cc:1428] Setting key a15779723595e225d0ba7b34f6732b25
I20260806 02:44:35.151774 25903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f: Bootstrap starting.
I20260806 02:44:35.154536 25903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:35.155459 25903 log.cc:824] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:35.157501 25903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f: No bootstrap required, opened a new log
Aug 06 02:44:35 dist-test-slave-tfpf krb5kdc[25788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:44:35.161551 25903 raft_consensus.cc:348] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } }
I20260806 02:44:35.161849 25903 raft_consensus.cc:374] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:35.161929 25903 raft_consensus.cc:729] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f839946eac7b45d58be4b9083fbb672f, State: Initialized, Role: FOLLOWER
I20260806 02:44:35.162418 25903 consensus_queue.cc:260] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } }
I20260806 02:44:35.162586 25903 raft_consensus.cc:388] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:35.162662 25903 raft_consensus.cc:482] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:35.162771 25903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:35.163937 25903 raft_consensus.cc:504] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } }
I20260806 02:44:35.164309 25903 leader_election.cc:302] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f839946eac7b45d58be4b9083fbb672f; no voters:
I20260806 02:44:35.164655 25903 leader_election.cc:288] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:35.164760 25909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:35.165035 25909 raft_consensus.cc:686] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [term 1 LEADER]: Becoming Leader. State: Replica: f839946eac7b45d58be4b9083fbb672f, State: Running, Role: LEADER
I20260806 02:44:35.165450 25909 consensus_queue.cc:237] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } }
I20260806 02:44:35.165985 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:35.151901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50573 (local address 127.0.0.1:45587)
0806 02:44:35.152346 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:35.152365 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:35.152406 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:44:35.153302 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:35.153326 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:35.153754 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:35.154112 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:35.154815 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:35.156173 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:35.157106 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:35.157789 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:35.157954 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:35.160409 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:35.160416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:35.160433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:35.160464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:35.162571 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:35.163176 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:35.163182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:35.163188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:35.163324 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:35.163658 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:35.163662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:35.163664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:35.164200 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:35.164396 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:35.164770 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:44:35.165007 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260806 02:44:35.165995 25903 sys_catalog.cc:564] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:35.167251 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f839946eac7b45d58be4b9083fbb672f. Latest consensus state: current_term: 1 leader_uuid: "f839946eac7b45d58be4b9083fbb672f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } } }
I20260806 02:44:35.167251 25910 sys_catalog.cc:455] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f839946eac7b45d58be4b9083fbb672f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f839946eac7b45d58be4b9083fbb672f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45587 } } }
I20260806 02:44:35.167403 25910 sys_catalog.cc:458] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:35.167403 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:35.168192 25923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:35.169346 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:35.171590 25923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:35.177462 25923 catalog_manager.cc:1347] Generated new cluster ID: 30cc26ffa56b41738fda875be38a05fe
I20260806 02:44:35.177767 25923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:35.180356 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:35.171359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58613 (local address 127.0.0.1:45587)
0806 02:44:35.171487 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:35.171491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:35.171745 (+ 254us) server_negotiation.cc:407] Connection header received
0806 02:44:35.171873 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:35.171876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:35.171916 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:35.172008 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:35.172655 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:35.173249 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:35.174324 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:35.174685 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:35.174800 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:35.176596 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:35.176600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:35.176602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:35.176627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:35.178480 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:35.179003 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:35.179007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:35.179009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:35.179099 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:35.179438 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:35.179441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:35.179443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:35.179655 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:35.179799 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:35.180066 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:44:35.180182 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:44:35.204141 25923 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:35.205387 25923 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:35.215931 25923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f839946eac7b45d58be4b9083fbb672f: Generated new TSK 0
I20260806 02:44:35.216748 25923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:35.227707 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25804
W20260806 02:44:35.238358 25786 connection.cc:535] client connection to 127.0.0.1:45587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45587): Connection reset by peer (error 104)
2026-08-06T02:44:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1614 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:35 dist-test-slave-tfpf krb5kdc[25944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:35 dist-test-slave-tfpf krb5kdc[25944](info): set up 2 sockets
Aug 06 02:44:35 dist-test-slave-tfpf krb5kdc[25944](info): commencing operation
krb5kdc: starting...
W20260806 02:44:36.447520 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:36 dist-test-slave-tfpf krb5kdc[25944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:36.577564 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:36.689155 25961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:36.689468 25961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:36.689545 25961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:36.693603 25961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:36.693686 25961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:36.693722 25961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:36.693785 25961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:36.693822 25961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:36.699117 25961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:36.700227 25961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:36.701327 25961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:36.709987 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:36.710024 25969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:36.709988 25971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:36.710400 25961 server_base.cc:1034] running on GCE node
I20260806 02:44:36.710968 25961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:36.712056 25961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:36.713213 25961 hybrid_clock.cc:648] HybridClock initialized: now 1785984276713200 us; error 24 us; skew 500 ppm
Aug 06 02:44:36 dist-test-slave-tfpf krb5kdc[25944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:36.715999 25961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:36.716920 25961 webserver.cc:460] Webserver started at http://127.0.0.1:33347/ using document root <none> and password file <none>
I20260806 02:44:36.717532 25961 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:36.717595 25961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:36.717806 25961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:36.719832 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "e6a1ee58875b4680829b875c819bdc94"
format_stamp: "Formatted at 2026-08-06 02:44:36 on dist-test-slave-tfpf"
server_key: "85f259084836cade4c6c9e2a48438f22"
server_key_iv: "c93c6e714364277286bd1a1b15b9d45d"
server_key_version: "encryptionkey@0"
I20260806 02:44:36.720403 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "e6a1ee58875b4680829b875c819bdc94"
format_stamp: "Formatted at 2026-08-06 02:44:36 on dist-test-slave-tfpf"
server_key: "85f259084836cade4c6c9e2a48438f22"
server_key_iv: "c93c6e714364277286bd1a1b15b9d45d"
server_key_version: "encryptionkey@0"
I20260806 02:44:36.724205 25961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:44:36.726663 25980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:36.727891 25961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:44:36.728035 25961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "e6a1ee58875b4680829b875c819bdc94"
format_stamp: "Formatted at 2026-08-06 02:44:36 on dist-test-slave-tfpf"
server_key: "85f259084836cade4c6c9e2a48438f22"
server_key_iv: "c93c6e714364277286bd1a1b15b9d45d"
server_key_version: "encryptionkey@0"
I20260806 02:44:36.728160 25961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:36.743687 25961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:36.746896 25961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:36.747107 25961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:36.754643 25961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45895
I20260806 02:44:36.754667 26042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45895 every 8 connection(s)
I20260806 02:44:36.755568 25961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:36.758266 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:36.763903 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 25961
I20260806 02:44:36.763924 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94: Bootstrap starting.
I20260806 02:44:36.764066 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:36.764411 29436 external_mini_cluster.cc:1428] Setting key afd87322621ce0f46646b4006269a508
I20260806 02:44:36.766655 26043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:36.767396 26043 log.cc:824] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:36.769538 26043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94: No bootstrap required, opened a new log
I20260806 02:44:36.773684 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:36.765775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54279 (local address 127.0.0.1:45895)
0806 02:44:36.766220 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:36.766235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:36.766268 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:44:36.766950 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:36.766968 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:36.767309 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:36.767645 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:36.768112 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:36.769184 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:36.770369 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:36.770823 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:36.770950 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:36.771888 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:36.771894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:36.771906 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:36.772301 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:36.772352 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:36.772374 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:36.772474 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:36.772711 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 02:44:36.772863 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260806 02:44:36.773864 26043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260806 02:44:36.774108 26043 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:36.774181 26043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a1ee58875b4680829b875c819bdc94, State: Initialized, Role: FOLLOWER
I20260806 02:44:36.774632 26043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260806 02:44:36.774793 26043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:36.774863 26043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:36.774959 26043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:36.776064 26043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260806 02:44:36.776449 26043 leader_election.cc:302] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a1ee58875b4680829b875c819bdc94; no voters:
I20260806 02:44:36.776799 26043 leader_election.cc:288] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:36.776913 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:36.776935 26050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:36.777235 26050 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [term 1 LEADER]: Becoming Leader. State: Replica: e6a1ee58875b4680829b875c819bdc94, State: Running, Role: LEADER
I20260806 02:44:36.777588 26050 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260806 02:44:36.778266 26043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:36.778708 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a1ee58875b4680829b875c819bdc94. Latest consensus state: current_term: 1 leader_uuid: "e6a1ee58875b4680829b875c819bdc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260806 02:44:36.778893 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:36.779181 26051 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a1ee58875b4680829b875c819bdc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a1ee58875b4680829b875c819bdc94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260806 02:44:36.779284 26051 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:36.779446 26064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:36.782738 26064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:36.784857 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:36.779674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37853 (local address 127.0.0.1:45895)
0806 02:44:36.779935 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:36.779940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:36.779990 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:44:36.780229 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:36.780234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:36.780294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:36.780351 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:36.780856 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:36.781661 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:36.782874 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:36.783219 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:36.783295 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:36.784340 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:36.784343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:36.784346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:36.784360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:36.784387 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:36.784400 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:36.784503 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:36.784689 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:44:36.784728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260806 02:44:36.788897 26064 catalog_manager.cc:1347] Generated new cluster ID: 4bd9b226ba794ce89f4b888b3d5f89de
I20260806 02:44:36.788972 26064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:36.802500 26064 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:36.803782 26064 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:36.808730 26064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6a1ee58875b4680829b875c819bdc94: Generated new TSK 0
I20260806 02:44:36.809557 26064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:36.814319 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 25961
W20260806 02:44:36.824155 25942 connection.cc:535] client connection to 127.0.0.1:45895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45895): Connection reset by peer (error 104)
2026-08-06T02:44:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:36 dist-test-slave-tfpf krb5kdc[26087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:36 dist-test-slave-tfpf krb5kdc[26087](info): set up 2 sockets
Aug 06 02:44:36 dist-test-slave-tfpf krb5kdc[26087](info): commencing operation
krb5kdc: starting...
W20260806 02:44:38.033680 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:38 dist-test-slave-tfpf krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:38.161288 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:38.277312 26104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:38.277613 26104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:38.277701 26104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:38.281734 26104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:38.281814 26104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:38.281852 26104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:38.281905 26104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:38.281944 26104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:38.287297 26104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:38.288373 26104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:38.289461 26104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:38.298456 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:38.298440 26112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:38.298640 26104 server_base.cc:1034] running on GCE node
W20260806 02:44:38.298661 26117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:38.299449 26104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:38.300632 26104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:38.301836 26104 hybrid_clock.cc:648] HybridClock initialized: now 1785984278301805 us; error 43 us; skew 500 ppm
Aug 06 02:44:38 dist-test-slave-tfpf krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:38.308583 26104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:38.309679 26104 webserver.cc:460] Webserver started at http://127.0.0.1:45209/ using document root <none> and password file <none>
I20260806 02:44:38.310299 26104 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:38.310365 26104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:38.310587 26104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:38.312716 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "88c5497f83a74b019d5bad6da3a6d665"
format_stamp: "Formatted at 2026-08-06 02:44:38 on dist-test-slave-tfpf"
server_key: "df57420656cadb6550ec3994a110922b"
server_key_iv: "68925bdfb38d568bedf4ca35cafd2bcc"
server_key_version: "encryptionkey@0"
I20260806 02:44:38.313344 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "88c5497f83a74b019d5bad6da3a6d665"
format_stamp: "Formatted at 2026-08-06 02:44:38 on dist-test-slave-tfpf"
server_key: "df57420656cadb6550ec3994a110922b"
server_key_iv: "68925bdfb38d568bedf4ca35cafd2bcc"
server_key_version: "encryptionkey@0"
I20260806 02:44:38.317198 26104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:44:38.319669 26124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:38.320976 26104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:38.321158 26104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "88c5497f83a74b019d5bad6da3a6d665"
format_stamp: "Formatted at 2026-08-06 02:44:38 on dist-test-slave-tfpf"
server_key: "df57420656cadb6550ec3994a110922b"
server_key_iv: "68925bdfb38d568bedf4ca35cafd2bcc"
server_key_version: "encryptionkey@0"
I20260806 02:44:38.321285 26104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:38.325932 26104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:38.329392 26104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:38.329598 26104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:38.336526 26104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40189
I20260806 02:44:38.336592 26187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40189 every 8 connection(s)
I20260806 02:44:38.337637 26104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:38.338490 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26104
I20260806 02:44:38.338652 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:38.338996 29436 external_mini_cluster.cc:1428] Setting key f57d682c7ce0f14f7ac613be8b3ab801
I20260806 02:44:38.341372 26188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:38.344388 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:38.340453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38613 (local address 127.0.0.1:40189)
0806 02:44:38.340905 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:38.340920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:38.340955 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:44:38.341698 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:38.341715 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:38.341982 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:38.342321 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:38.343280 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:38.343284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:38.343294 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:38.343301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:38.343338 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:38.343354 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:38.343505 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:38.343585 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:44:38.343738 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260806 02:44:38.347415 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:38.348021 26188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665: Bootstrap starting.
I20260806 02:44:38.350770 26188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:38.351152 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:38.349554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34199 (local address 127.0.0.1:40189)
0806 02:44:38.349682 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:38.349686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:38.349851 (+ 165us) server_negotiation.cc:407] Connection header received
0806 02:44:38.349972 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:38.349974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:38.350001 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:38.350065 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:38.350751 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:38.350754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:38.350756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:38.350765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:38.350786 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:38.350796 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:38.350895 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:38.351014 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:44:38.351045 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:44:38.351704 26188 log.cc:824] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:38.353727 26188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665: No bootstrap required, opened a new log
I20260806 02:44:38.357494 26188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } }
I20260806 02:44:38.357759 26188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:38.357846 26188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c5497f83a74b019d5bad6da3a6d665, State: Initialized, Role: FOLLOWER
I20260806 02:44:38.358286 26188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } }
I20260806 02:44:38.358451 26188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:38.358553 26188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:38.358685 26188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:38.359602 26188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } }
I20260806 02:44:38.359954 26188 leader_election.cc:302] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c5497f83a74b019d5bad6da3a6d665; no voters:
I20260806 02:44:38.360347 26188 leader_election.cc:288] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:38.360427 26201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:38.360685 26201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [term 1 LEADER]: Becoming Leader. State: Replica: 88c5497f83a74b019d5bad6da3a6d665, State: Running, Role: LEADER
I20260806 02:44:38.361002 26201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } }
I20260806 02:44:38.361749 26188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:38.362854 26204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c5497f83a74b019d5bad6da3a6d665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } } }
I20260806 02:44:38.362876 26202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c5497f83a74b019d5bad6da3a6d665. Latest consensus state: current_term: 1 leader_uuid: "88c5497f83a74b019d5bad6da3a6d665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c5497f83a74b019d5bad6da3a6d665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40189 } } }
I20260806 02:44:38.363075 26204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:38.363176 26202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:38.363873 26216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:38.366605 26216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:38.372112 26216 catalog_manager.cc:1347] Generated new cluster ID: 2efa50b8615b4c979459a36a312b3ee9
I20260806 02:44:38.372185 26216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:38.382557 26216 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:38.382965 26216 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:38.388130 26216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c5497f83a74b019d5bad6da3a6d665: Generated new TSK 0
I20260806 02:44:38.388914 26216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:38.396293 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26104
2026-08-06T02:44:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:38 dist-test-slave-tfpf krb5kdc[26231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:38 dist-test-slave-tfpf krb5kdc[26231](info): set up 2 sockets
Aug 06 02:44:38 dist-test-slave-tfpf krb5kdc[26231](info): commencing operation
krb5kdc: starting...
W20260806 02:44:39.615649 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:39 dist-test-slave-tfpf krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:39.746217 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:39.864171 26251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:39.864499 26251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:39.864585 26251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:39.868639 26251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:39.868741 26251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:39.868785 26251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:39.868853 26251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:39.868898 26251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:39.874228 26251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:39.875342 26251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:39.876467 26251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:39.885365 26258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:39.885356 26257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:39.885496 26260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:39.885826 26251 server_base.cc:1034] running on GCE node
I20260806 02:44:39.886466 26251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:39.887552 26251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:39.888804 26251 hybrid_clock.cc:648] HybridClock initialized: now 1785984279888773 us; error 42 us; skew 500 ppm
Aug 06 02:44:39 dist-test-slave-tfpf krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:39.891784 26251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:39.892654 26251 webserver.cc:460] Webserver started at http://127.0.0.1:45189/ using document root <none> and password file <none>
I20260806 02:44:39.893262 26251 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:39.893334 26251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:39.893514 26251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:39.895596 26251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "2a944ed27c85441698aa0c54cc24daa1"
format_stamp: "Formatted at 2026-08-06 02:44:39 on dist-test-slave-tfpf"
server_key: "63c267ce98ca2dccc7ade6a8604855f3"
server_key_iv: "b49c33296a6afc643b3c4af0a85d15e0"
server_key_version: "encryptionkey@0"
I20260806 02:44:39.896181 26251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "2a944ed27c85441698aa0c54cc24daa1"
format_stamp: "Formatted at 2026-08-06 02:44:39 on dist-test-slave-tfpf"
server_key: "63c267ce98ca2dccc7ade6a8604855f3"
server_key_iv: "b49c33296a6afc643b3c4af0a85d15e0"
server_key_version: "encryptionkey@0"
I20260806 02:44:39.900110 26251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 02:44:39.902643 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:39.903913 26251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:44:39.904085 26251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "2a944ed27c85441698aa0c54cc24daa1"
format_stamp: "Formatted at 2026-08-06 02:44:39 on dist-test-slave-tfpf"
server_key: "63c267ce98ca2dccc7ade6a8604855f3"
server_key_iv: "b49c33296a6afc643b3c4af0a85d15e0"
server_key_version: "encryptionkey@0"
I20260806 02:44:39.904210 26251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:39.931824 26251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:39.935176 26251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:39.935389 26251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:39.943120 26251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40953
I20260806 02:44:39.943269 26336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40953 every 8 connection(s)
I20260806 02:44:39.944238 26251 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:39.947249 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:39.953246 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1: Bootstrap starting.
I20260806 02:44:39.954089 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26251
I20260806 02:44:39.954279 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:39.954560 29436 external_mini_cluster.cc:1428] Setting key 49e84de4b2e007e6ed87cc824a627fd9
I20260806 02:44:39.956220 26337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:39.957192 26337 log.cc:824] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:39.959489 26337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1: No bootstrap required, opened a new log
I20260806 02:44:39.963510 26337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260806 02:44:39.963809 26337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:39.963871 26337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944ed27c85441698aa0c54cc24daa1, State: Initialized, Role: FOLLOWER
Aug 06 02:44:39 dist-test-slave-tfpf krb5kdc[26231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:44:39.964401 26337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260806 02:44:39.964591 26337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:39.964658 26337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:39.964773 26337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:39.965973 26337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260806 02:44:39.966374 26337 leader_election.cc:302] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a944ed27c85441698aa0c54cc24daa1; no voters:
I20260806 02:44:39.966753 26337 leader_election.cc:288] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:39.966821 26345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:39.967034 26345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a944ed27c85441698aa0c54cc24daa1, State: Running, Role: LEADER
I20260806 02:44:39.967383 26345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260806 02:44:39.968119 26337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:39.968583 26345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a944ed27c85441698aa0c54cc24daa1. Latest consensus state: current_term: 1 leader_uuid: "2a944ed27c85441698aa0c54cc24daa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } } }
I20260806 02:44:39.969426 26345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:39.969310 26346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a944ed27c85441698aa0c54cc24daa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a944ed27c85441698aa0c54cc24daa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } } }
I20260806 02:44:39.969741 26346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:39.970767 26358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:39.971511 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:39.955796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40463 (local address 127.0.0.1:40953)
0806 02:44:39.956301 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:39.956322 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:39.956366 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:44:39.957329 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:39.957354 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:39.957792 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:39.958142 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:39.958858 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:39.960306 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:39.961274 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:39.962005 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:39.962191 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:39.965002 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:39.965007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:39.965021 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:39.965050 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:39.967201 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:39.967783 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:39.967789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:39.967797 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:39.967917 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:39.968267 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:39.968269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:39.968271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:39.968798 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:39.969808 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:39.969971 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:44:39.970346 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260806 02:44:39.974344 26358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:39.974489 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:39.980413 26358 catalog_manager.cc:1347] Generated new cluster ID: d9c5eaaa05534dbd8cec03105eef71a3
I20260806 02:44:39.980485 26358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:39.986070 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:39.976907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53647 (local address 127.0.0.1:40953)
0806 02:44:39.977204 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:39.977209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:39.977222 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:44:39.977259 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:39.977262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:39.977307 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:39.977391 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:39.978267 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:39.978878 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:39.980149 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:39.980599 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:39.980700 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:39.982601 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:39.982605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:39.982608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:39.982634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:39.984306 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:39.984807 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:39.984810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:39.984812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:39.984892 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:39.985216 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:39.985219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:39.985220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:39.985416 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:39.985531 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:39.985801 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:44:39.985917 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260806 02:44:40.003084 26358 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:40.004447 26358 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:40.014289 26358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a944ed27c85441698aa0c54cc24daa1: Generated new TSK 0
I20260806 02:44:40.015234 26358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:40.022130 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26251
2026-08-06T02:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1623 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:40 dist-test-slave-tfpf krb5kdc[26383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:40 dist-test-slave-tfpf krb5kdc[26383](info): set up 2 sockets
Aug 06 02:44:40 dist-test-slave-tfpf krb5kdc[26383](info): commencing operation
krb5kdc: starting...
W20260806 02:44:41.255061 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:41 dist-test-slave-tfpf krb5kdc[26383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:41.396879 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:41.511826 26402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:41.512176 26402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:41.512267 26402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:41.516322 26402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:41.516435 26402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:41.516479 26402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:41.516541 26402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:41.516588 26402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:41.522014 26402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:41.523180 26402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:41.524279 26402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:41.533530 26416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:41.533541 26414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:41.533635 26402 server_base.cc:1034] running on GCE node
W20260806 02:44:41.533541 26412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:41.534355 26402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:41.535449 26402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:41.536643 26402 hybrid_clock.cc:648] HybridClock initialized: now 1785984281536629 us; error 33 us; skew 500 ppm
Aug 06 02:44:41 dist-test-slave-tfpf krb5kdc[26383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:41.539588 26402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:41.540490 26402 webserver.cc:460] Webserver started at http://127.0.0.1:39739/ using document root <none> and password file <none>
I20260806 02:44:41.541026 26402 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:41.541154 26402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:41.541347 26402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:41.543385 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "61a8476083a04495b15707ce6920d248"
format_stamp: "Formatted at 2026-08-06 02:44:41 on dist-test-slave-tfpf"
server_key: "922dd8c0548f04e70bc9f93b5733cf04"
server_key_iv: "c5e7603eb082bb86e9e6cb79c3cb501f"
server_key_version: "encryptionkey@0"
I20260806 02:44:41.543912 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "61a8476083a04495b15707ce6920d248"
format_stamp: "Formatted at 2026-08-06 02:44:41 on dist-test-slave-tfpf"
server_key: "922dd8c0548f04e70bc9f93b5733cf04"
server_key_iv: "c5e7603eb082bb86e9e6cb79c3cb501f"
server_key_version: "encryptionkey@0"
I20260806 02:44:41.547763 26402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:44:41.550303 26426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:41.551695 26402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:44:41.551805 26402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "61a8476083a04495b15707ce6920d248"
format_stamp: "Formatted at 2026-08-06 02:44:41 on dist-test-slave-tfpf"
server_key: "922dd8c0548f04e70bc9f93b5733cf04"
server_key_iv: "c5e7603eb082bb86e9e6cb79c3cb501f"
server_key_version: "encryptionkey@0"
I20260806 02:44:41.551894 26402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:41.568476 26402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:41.571758 26402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:41.571918 26402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:41.579409 26402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36055
I20260806 02:44:41.579432 26502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36055 every 8 connection(s)
I20260806 02:44:41.580432 26402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:41.583388 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:41.585026 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26402
I20260806 02:44:41.585247 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:41.585562 29436 external_mini_cluster.cc:1428] Setting key b807f2ea7ea52ecd21e3d3117d19e52e
I20260806 02:44:41.591171 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248: Bootstrap starting.
I20260806 02:44:41.593894 26503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:41.594673 26503 log.cc:824] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:41.596325 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:41.587552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38889 (local address 127.0.0.1:36055)
0806 02:44:41.588185 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:41.588204 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:41.588252 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:44:41.589175 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:41.589206 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:41.589654 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:41.590030 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:41.590466 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:41.591884 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:41.592850 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:41.593412 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:41.593595 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:41.594519 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:41.594527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:41.594546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:41.594986 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:41.595044 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:41.595075 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:41.595185 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:41.595304 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:44:41.595535 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":150,"threads_started":1}
I20260806 02:44:41.596768 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248: No bootstrap required, opened a new log
I20260806 02:44:41.599504 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:41.601560 26503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } }
I20260806 02:44:41.601876 26503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:41.602002 26503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a8476083a04495b15707ce6920d248, State: Initialized, Role: FOLLOWER
I20260806 02:44:41.602562 26503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } }
I20260806 02:44:41.602764 26503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:41.602844 26503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:41.602962 26503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:41.604120 26503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } }
I20260806 02:44:41.604506 26503 leader_election.cc:302] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a8476083a04495b15707ce6920d248; no voters:
I20260806 02:44:41.604914 26503 leader_election.cc:288] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:41.604986 26521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:41.605464 26521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [term 1 LEADER]: Becoming Leader. State: Replica: 61a8476083a04495b15707ce6920d248, State: Running, Role: LEADER
I20260806 02:44:41.605948 26521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } }
I20260806 02:44:41.606640 26503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:41.606660 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:41.602122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57419 (local address 127.0.0.1:36055)
0806 02:44:41.602272 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:41.602279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:41.602374 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:44:41.602513 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:41.602516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:41.602568 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:41.602637 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:41.603014 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:41.603577 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:41.604764 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:41.605145 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:41.605298 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:41.606231 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:41.606234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:41.606236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:41.606248 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:41.606274 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:41.606285 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:41.606371 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:41.606467 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:44:41.606522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:41.607337 26521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a8476083a04495b15707ce6920d248. Latest consensus state: current_term: 1 leader_uuid: "61a8476083a04495b15707ce6920d248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } } }
I20260806 02:44:41.607602 26521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:41.607838 26524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a8476083a04495b15707ce6920d248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a8476083a04495b15707ce6920d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36055 } } }
I20260806 02:44:41.607986 26524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:41.608222 26530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:41.611948 26530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:41.618448 26530 catalog_manager.cc:1347] Generated new cluster ID: c16b51f9c4564f4393104d132ed4683e
I20260806 02:44:41.618549 26530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:41.631608 26530 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:41.632925 26530 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:41.641561 26530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a8476083a04495b15707ce6920d248: Generated new TSK 0
I20260806 02:44:41.642334 26530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:41.651952 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26402
2026-08-06T02:44:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:41 dist-test-slave-tfpf krb5kdc[26550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:41 dist-test-slave-tfpf krb5kdc[26550](info): set up 2 sockets
Aug 06 02:44:41 dist-test-slave-tfpf krb5kdc[26550](info): commencing operation
krb5kdc: starting...
W20260806 02:44:42.894456 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:42 dist-test-slave-tfpf krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:43.050416 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:43.172870 26566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:43.173156 26566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:43.173200 26566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:43.177335 26566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:43.177405 26566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:43.177425 26566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:43.177443 26566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:43.177459 26566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:43.183198 26566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:43.184307 26566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:43.185367 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:43.194020 26577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:43.194027 26575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:43.194113 26574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:43.194509 26566 server_base.cc:1034] running on GCE node
I20260806 02:44:43.195109 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:43.196310 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:43.197489 26566 hybrid_clock.cc:648] HybridClock initialized: now 1785984283197472 us; error 38 us; skew 500 ppm
Aug 06 02:44:43 dist-test-slave-tfpf krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:43.200340 26566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:43.201324 26566 webserver.cc:460] Webserver started at http://127.0.0.1:44503/ using document root <none> and password file <none>
I20260806 02:44:43.201900 26566 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:43.201994 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:43.202243 26566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:43.204344 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "aa14330caca14697bbd4a93aed01f8a5"
format_stamp: "Formatted at 2026-08-06 02:44:43 on dist-test-slave-tfpf"
server_key: "a22c3c543509a3f6b7e186d8fc3de681"
server_key_iv: "c737ef75bbd1794e3ae4719579dafe15"
server_key_version: "encryptionkey@0"
I20260806 02:44:43.204939 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "aa14330caca14697bbd4a93aed01f8a5"
format_stamp: "Formatted at 2026-08-06 02:44:43 on dist-test-slave-tfpf"
server_key: "a22c3c543509a3f6b7e186d8fc3de681"
server_key_iv: "c737ef75bbd1794e3ae4719579dafe15"
server_key_version: "encryptionkey@0"
I20260806 02:44:43.208921 26566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:44:43.211532 26586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:43.212946 26566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:44:43.213155 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "aa14330caca14697bbd4a93aed01f8a5"
format_stamp: "Formatted at 2026-08-06 02:44:43 on dist-test-slave-tfpf"
server_key: "a22c3c543509a3f6b7e186d8fc3de681"
server_key_iv: "c737ef75bbd1794e3ae4719579dafe15"
server_key_version: "encryptionkey@0"
I20260806 02:44:43.213285 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:43.218263 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:43.221495 26566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:43.221694 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:43.228892 26566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41765
I20260806 02:44:43.228921 26661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41765 every 8 connection(s)
I20260806 02:44:43.229900 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:43.232890 26663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:43.238536 26663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5: Bootstrap starting.
I20260806 02:44:43.238554 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26566
I20260806 02:44:43.238703 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:43.239012 29436 external_mini_cluster.cc:1428] Setting key 8806167e1f2389dc9dcbacf2d617ccab
I20260806 02:44:43.241237 26663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:43.241966 26663 log.cc:824] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:43.243953 26663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5: No bootstrap required, opened a new log
I20260806 02:44:43.245994 26668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:43.240378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43009 (local address 127.0.0.1:41765)
0806 02:44:43.240845 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:43.240860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:43.240895 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:44:43.241640 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:43.241659 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:43.241940 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:43.242306 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:43.244446 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:43.244451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:43.244466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:43.244480 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:43.244552 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:43.244580 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:43.244715 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:43.244881 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:44:43.245037 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
I20260806 02:44:43.248493 26663 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260806 02:44:43.248775 26663 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:43.248854 26663 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa14330caca14697bbd4a93aed01f8a5, State: Initialized, Role: FOLLOWER
I20260806 02:44:43.248941 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:43.249346 26663 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260806 02:44:43.249522 26663 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:43.249593 26663 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:43.249691 26663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:43.251006 26663 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260806 02:44:43.251413 26663 leader_election.cc:302] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa14330caca14697bbd4a93aed01f8a5; no voters:
I20260806 02:44:43.251844 26663 leader_election.cc:288] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:43.252019 26677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:43.252302 26677 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [term 1 LEADER]: Becoming Leader. State: Replica: aa14330caca14697bbd4a93aed01f8a5, State: Running, Role: LEADER
I20260806 02:44:43.253103 26677 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260806 02:44:43.253278 26663 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:43.253693 26668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:43.251465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33053 (local address 127.0.0.1:41765)
0806 02:44:43.251627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:43.251632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:43.251892 (+ 260us) server_negotiation.cc:407] Connection header received
0806 02:44:43.252022 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:43.252027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:43.252076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:43.252184 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:43.253163 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:43.253167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:43.253170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:43.253184 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:43.253221 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:43.253240 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:43.253341 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:43.253521 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 02:44:43.253562 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:44:43.255259 26678 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa14330caca14697bbd4a93aed01f8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } } }
I20260806 02:44:43.255299 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa14330caca14697bbd4a93aed01f8a5. Latest consensus state: current_term: 1 leader_uuid: "aa14330caca14697bbd4a93aed01f8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa14330caca14697bbd4a93aed01f8a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } } }
I20260806 02:44:43.255405 26678 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:43.255419 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:43.255900 26697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:43.259518 26697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:43.265421 26697 catalog_manager.cc:1347] Generated new cluster ID: 182d4524e36b49dd8d53a0b41e4dbbf5
I20260806 02:44:43.265507 26697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:43.290987 26697 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:43.291384 26697 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:43.301620 26697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa14330caca14697bbd4a93aed01f8a5: Generated new TSK 0
I20260806 02:44:43.302394 26697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:43.314332 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26566
2026-08-06T02:44:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:43 dist-test-slave-tfpf krb5kdc[26706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:43 dist-test-slave-tfpf krb5kdc[26706](info): set up 2 sockets
Aug 06 02:44:43 dist-test-slave-tfpf krb5kdc[26706](info): commencing operation
krb5kdc: starting...
W20260806 02:44:44.547681 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:44 dist-test-slave-tfpf krb5kdc[26706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:44.689640 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:44.812745 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:44.813135 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:44.813243 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:44.817415 26724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:44.817512 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:44.817545 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:44.817595 26724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:44.817646 26724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:44.823344 26724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:44.824595 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:44.825841 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:44.835579 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:44.835624 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:44.835624 26734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:44.835688 26724 server_base.cc:1034] running on GCE node
I20260806 02:44:44.836515 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:44.837821 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:44.838994 26724 hybrid_clock.cc:648] HybridClock initialized: now 1785984284838972 us; error 41 us; skew 500 ppm
Aug 06 02:44:44 dist-test-slave-tfpf krb5kdc[26706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:44.842267 26724 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:44.843235 26724 webserver.cc:460] Webserver started at http://127.0.0.1:35023/ using document root <none> and password file <none>
I20260806 02:44:44.843827 26724 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:44.843894 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:44.844125 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:44.846380 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc"
format_stamp: "Formatted at 2026-08-06 02:44:44 on dist-test-slave-tfpf"
server_key: "e52c381f3375898b953689c93d93ee73"
server_key_iv: "4a54edb4fb45689d3c1ee5a0775f8551"
server_key_version: "encryptionkey@0"
I20260806 02:44:44.846982 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc"
format_stamp: "Formatted at 2026-08-06 02:44:44 on dist-test-slave-tfpf"
server_key: "e52c381f3375898b953689c93d93ee73"
server_key_iv: "4a54edb4fb45689d3c1ee5a0775f8551"
server_key_version: "encryptionkey@0"
I20260806 02:44:44.851219 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:44:44.854092 26745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:44.855580 26724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:44:44.855822 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc"
format_stamp: "Formatted at 2026-08-06 02:44:44 on dist-test-slave-tfpf"
server_key: "e52c381f3375898b953689c93d93ee73"
server_key_iv: "4a54edb4fb45689d3c1ee5a0775f8551"
server_key_version: "encryptionkey@0"
I20260806 02:44:44.855981 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:44.878342 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:44.881929 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:44.882225 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:44.890125 26724 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41007
I20260806 02:44:44.890151 26809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41007 every 8 connection(s)
I20260806 02:44:44.891248 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:44.894405 26811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:44.898368 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26724
I20260806 02:44:44.898533 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:44.898839 29436 external_mini_cluster.cc:1428] Setting key cf061235195fa3a1bf1ca3e317b9c459
I20260806 02:44:44.901321 26811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: Bootstrap starting.
I20260806 02:44:44.904332 26811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:44.905301 26811 log.cc:824] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:44.907696 26811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: No bootstrap required, opened a new log
Aug 06 02:44:44 dist-test-slave-tfpf krb5kdc[26706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:44:44.911932 26811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } }
I20260806 02:44:44.912256 26811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:44.912328 26811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa7edb21b794c5c9e6aa9f3983e64bc, State: Initialized, Role: FOLLOWER
I20260806 02:44:44.912844 26811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } }
I20260806 02:44:44.913017 26811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:44.913111 26811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:44.913234 26811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:44.914446 26811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } }
I20260806 02:44:44.914826 26811 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa7edb21b794c5c9e6aa9f3983e64bc; no voters:
I20260806 02:44:44.915197 26811 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:44.915315 26817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:44.915587 26817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [term 1 LEADER]: Becoming Leader. State: Replica: 3aa7edb21b794c5c9e6aa9f3983e64bc, State: Running, Role: LEADER
I20260806 02:44:44.915968 26817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } }
I20260806 02:44:44.916523 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:44.900446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:35619 (local address 127.0.0.1:41007)
0806 02:44:44.900918 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:44.900940 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:44.900987 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:44:44.901952 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:44.901978 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:44.902445 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:44.902869 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:44.903545 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:44.905289 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:44.906477 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:44.907189 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:44.907377 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:44.910277 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:44.910283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:44.910300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:44.910335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:44.912991 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:44.913675 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:44.913681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:44.913687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:44.913826 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:44.914190 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:44.914194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:44.914196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:44.914759 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:44.914966 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:44.915334 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 02:44:44.915575 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260806 02:44:44.916539 26811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:44.917434 26818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa7edb21b794c5c9e6aa9f3983e64bc. Latest consensus state: current_term: 1 leader_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } } }
I20260806 02:44:44.917551 26818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:44.918219 26819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa7edb21b794c5c9e6aa9f3983e64bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41007 } } }
I20260806 02:44:44.918399 26819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:44.919958 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 02:44:44.921962 26840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:44:44.922043 26840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:44:44.922145 26833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:44.923018 26833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:44.929627 26833 catalog_manager.cc:1347] Generated new cluster ID: 3ebe11fc33da43efb50bbaff70599b44
I20260806 02:44:44.929695 26833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:44.934676 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:44.922292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:40773 (local address 127.0.0.1:41007)
0806 02:44:44.922410 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:44.922414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:44.922577 (+ 163us) server_negotiation.cc:407] Connection header received
0806 02:44:44.922729 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:44.922733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:44.922783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:44.922868 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:44.923609 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:44.924124 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:44.924965 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:44.925368 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:44.925483 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:44.927220 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:44.927225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:44.927232 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:44.927254 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:44.928894 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:44.933298 (+ 4404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:44.933302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:44.933303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:44.933394 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:44.933738 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:44.933741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:44.933743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:44.933925 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:44.934090 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:44.934374 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:44:44.934475 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:44:44.945652 26833 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:44.946889 26833 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:44.953970 26833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aa7edb21b794c5c9e6aa9f3983e64bc: Generated new TSK 0
I20260806 02:44:44.954905 26833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:44.963593 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26724
W20260806 02:44:44.979341 26702 connection.cc:535] client connection to 127.0.0.1:41007 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41007): Connection reset by peer (error 104)
2026-08-06T02:44:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1654 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:45 dist-test-slave-tfpf krb5kdc[26852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:45 dist-test-slave-tfpf krb5kdc[26852](info): set up 2 sockets
Aug 06 02:44:45 dist-test-slave-tfpf krb5kdc[26852](info): commencing operation
krb5kdc: starting...
W20260806 02:44:46.200402 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:46 dist-test-slave-tfpf krb5kdc[26852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:46.340822 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:46.461352 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:46.461691 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:46.461791 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:46.465866 26870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:46.465973 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:46.466013 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:46.466048 26870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:46.466076 26870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:46.471719 26870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:46.472934 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:46.474054 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:46.483135 26879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:46.483174 26882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:46.483287 26870 server_base.cc:1034] running on GCE node
W20260806 02:44:46.483178 26878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:46.484011 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:46.485181 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:46.486346 26870 hybrid_clock.cc:648] HybridClock initialized: now 1785984286486326 us; error 32 us; skew 500 ppm
Aug 06 02:44:46 dist-test-slave-tfpf krb5kdc[26852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:46.489243 26870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:46.490185 26870 webserver.cc:460] Webserver started at http://127.0.0.1:44327/ using document root <none> and password file <none>
I20260806 02:44:46.490774 26870 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:46.490869 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:46.491142 26870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:46.493309 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "ad26451de70645999c4702a2db27ea20"
format_stamp: "Formatted at 2026-08-06 02:44:46 on dist-test-slave-tfpf"
server_key: "20a31ab2f060650e8d7cf36685deccb6"
server_key_iv: "480026ca20c4d4db1df8b68aa3614ffb"
server_key_version: "encryptionkey@0"
I20260806 02:44:46.493888 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "ad26451de70645999c4702a2db27ea20"
format_stamp: "Formatted at 2026-08-06 02:44:46 on dist-test-slave-tfpf"
server_key: "20a31ab2f060650e8d7cf36685deccb6"
server_key_iv: "480026ca20c4d4db1df8b68aa3614ffb"
server_key_version: "encryptionkey@0"
I20260806 02:44:46.497678 26870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:44:46.500142 26889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:46.501461 26870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:46.501624 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "ad26451de70645999c4702a2db27ea20"
format_stamp: "Formatted at 2026-08-06 02:44:46 on dist-test-slave-tfpf"
server_key: "20a31ab2f060650e8d7cf36685deccb6"
server_key_iv: "480026ca20c4d4db1df8b68aa3614ffb"
server_key_version: "encryptionkey@0"
I20260806 02:44:46.501747 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:46.522812 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:46.526278 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:46.526539 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:46.534858 26870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39347
I20260806 02:44:46.534881 26955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39347 every 8 connection(s)
I20260806 02:44:46.535902 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:46.538810 26956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:46.539593 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 26870
I20260806 02:44:46.539743 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:46.540056 29436 external_mini_cluster.cc:1428] Setting key 0a893098da4a4f24a756d94caff4e69c
I20260806 02:44:46.547528 26956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20: Bootstrap starting.
I20260806 02:44:46.550412 26956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:46.551187 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:46.541598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:43405 (local address 127.0.0.1:39347)
0806 02:44:46.542098 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:46.542120 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:46.542169 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:44:46.543214 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:46.543310 (+ 96us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:46.543800 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:46.544218 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:46.544655 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:46.546366 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:46.547276 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:46.548001 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:46.548226 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:46.549204 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:46.549210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:46.549225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:46.549701 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:46.549770 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:46.549801 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:46.549935 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:46.550115 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 02:44:46.550252 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260806 02:44:46.551295 26956 log.cc:824] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:46.553455 26956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20: No bootstrap required, opened a new log
I20260806 02:44:46.554345 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:46.557507 26956 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 02:44:46.557781 26956 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:46.557860 26956 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad26451de70645999c4702a2db27ea20, State: Initialized, Role: FOLLOWER
I20260806 02:44:46.558362 26956 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 02:44:46.558537 26956 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:46.558598 26956 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:46.558702 26956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:46.559851 26956 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 02:44:46.560247 26956 leader_election.cc:302] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad26451de70645999c4702a2db27ea20; no voters:
I20260806 02:44:46.560626 26956 leader_election.cc:288] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:46.560708 26967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:46.560926 26967 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [term 1 LEADER]: Becoming Leader. State: Replica: ad26451de70645999c4702a2db27ea20, State: Running, Role: LEADER
I20260806 02:44:46.561302 26967 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 02:44:46.561789 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:46.556605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:46239 (local address 127.0.0.1:39347)
0806 02:44:46.556776 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:46.556781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:46.556952 (+ 171us) server_negotiation.cc:407] Connection header received
0806 02:44:46.557169 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:46.557174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:46.557224 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:46.557297 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:46.557608 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:46.558299 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:46.559426 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:46.559720 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:46.559810 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:46.560904 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:46.560916 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:46.560919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:46.560936 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:46.560967 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:46.560990 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:46.561250 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:46.561646 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 02:44:46.561685 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:44:46.561982 26956 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:46.563097 26968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad26451de70645999c4702a2db27ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260806 02:44:46.563248 26968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:46.563709 26983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:46.564035 26969 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad26451de70645999c4702a2db27ea20. Latest consensus state: current_term: 1 leader_uuid: "ad26451de70645999c4702a2db27ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad26451de70645999c4702a2db27ea20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260806 02:44:46.564209 26969 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:46.566864 26983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:46.572904 26983 catalog_manager.cc:1347] Generated new cluster ID: e023b70d48794f1fb877c147b82930e9
I20260806 02:44:46.572989 26983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:46.588270 26983 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:46.589646 26983 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:46.597602 26983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad26451de70645999c4702a2db27ea20: Generated new TSK 0
I20260806 02:44:46.598440 26983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:46.602671 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 26870
W20260806 02:44:46.616154 26850 connection.cc:535] client connection to 127.0.0.1:39347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39347): Connection reset by peer (error 104)
2026-08-06T02:44:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:46 dist-test-slave-tfpf krb5kdc[26999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:46 dist-test-slave-tfpf krb5kdc[26999](info): set up 2 sockets
Aug 06 02:44:46 dist-test-slave-tfpf krb5kdc[26999](info): commencing operation
krb5kdc: starting...
W20260806 02:44:47.865121 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:47 dist-test-slave-tfpf krb5kdc[26999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:48.000247 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:48.121912 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:48.122217 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:48.122298 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:48.126475 27022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:48.126555 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:48.126576 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:48.126593 27022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:48.126609 27022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:48.132403 27022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:48.133665 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:48.134881 27022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:48.144855 27029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:48.144917 27032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:48.144913 27030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:48.145134 27022 server_base.cc:1034] running on GCE node
I20260806 02:44:48.145932 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:48.147168 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:48.148375 27022 hybrid_clock.cc:648] HybridClock initialized: now 1785984288148355 us; error 35 us; skew 500 ppm
Aug 06 02:44:48 dist-test-slave-tfpf krb5kdc[26999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:48.151670 27022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:48.152684 27022 webserver.cc:460] Webserver started at http://127.0.0.1:45495/ using document root <none> and password file <none>
I20260806 02:44:48.153384 27022 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:48.153489 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:48.153766 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:48.156032 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "de74ebcc05134eb5aa30f1b2a68165b6"
format_stamp: "Formatted at 2026-08-06 02:44:48 on dist-test-slave-tfpf"
server_key: "c1aebe855849f0b775de6d7da913fa39"
server_key_iv: "5acbc07a40b6359930924732c200f958"
server_key_version: "encryptionkey@0"
I20260806 02:44:48.156692 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "de74ebcc05134eb5aa30f1b2a68165b6"
format_stamp: "Formatted at 2026-08-06 02:44:48 on dist-test-slave-tfpf"
server_key: "c1aebe855849f0b775de6d7da913fa39"
server_key_iv: "5acbc07a40b6359930924732c200f958"
server_key_version: "encryptionkey@0"
I20260806 02:44:48.160945 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:44:48.163802 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:48.165194 27022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:44:48.165354 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "de74ebcc05134eb5aa30f1b2a68165b6"
format_stamp: "Formatted at 2026-08-06 02:44:48 on dist-test-slave-tfpf"
server_key: "c1aebe855849f0b775de6d7da913fa39"
server_key_iv: "5acbc07a40b6359930924732c200f958"
server_key_version: "encryptionkey@0"
I20260806 02:44:48.165484 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:48.170733 27022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:48.174440 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:48.174669 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:48.182672 27022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34473
I20260806 02:44:48.182760 27111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34473 every 8 connection(s)
I20260806 02:44:48.183719 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:48.187229 27113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:48.188737 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27022
I20260806 02:44:48.188915 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:48.189281 29436 external_mini_cluster.cc:1428] Setting key eb8494af7263da9d5ff447578339d013
I20260806 02:44:48.194775 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: Bootstrap starting.
I20260806 02:44:48.195498 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:48.190741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:38383 (local address 127.0.0.1:34473)
0806 02:44:48.191226 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:48.191254 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:48.191299 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:44:48.192215 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:48.192241 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:48.192634 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:48.193051 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:48.193930 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:48.193936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:48.193961 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:48.193974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:48.194058 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:48.194088 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:48.194237 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:48.194309 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:44:48.194476 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260806 02:44:48.197734 27113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:48.198679 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:48.198722 27113 log.cc:824] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:48.200958 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: No bootstrap required, opened a new log
I20260806 02:44:48.203212 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:48.201151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:47217 (local address 127.0.0.1:34473)
0806 02:44:48.201295 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:48.201300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:48.201572 (+ 272us) server_negotiation.cc:407] Connection header received
0806 02:44:48.201692 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:48.201695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:48.201732 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:48.201828 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:48.202755 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:48.202759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:48.202762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:48.202777 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:48.202809 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:48.202823 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:48.202922 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:48.203012 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 02:44:48.203069 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:44:48.205804 27113 raft_consensus.cc:348] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260806 02:44:48.206046 27113 raft_consensus.cc:374] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:48.206085 27113 raft_consensus.cc:729] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de74ebcc05134eb5aa30f1b2a68165b6, State: Initialized, Role: FOLLOWER
I20260806 02:44:48.206552 27113 consensus_queue.cc:260] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260806 02:44:48.206712 27113 raft_consensus.cc:388] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:48.206765 27113 raft_consensus.cc:482] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:48.206845 27113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:48.207861 27113 raft_consensus.cc:504] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260806 02:44:48.208293 27113 leader_election.cc:302] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de74ebcc05134eb5aa30f1b2a68165b6; no voters:
I20260806 02:44:48.208716 27113 leader_election.cc:288] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:48.208856 27124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:48.209157 27124 raft_consensus.cc:686] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [term 1 LEADER]: Becoming Leader. State: Replica: de74ebcc05134eb5aa30f1b2a68165b6, State: Running, Role: LEADER
I20260806 02:44:48.209549 27124 consensus_queue.cc:237] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260806 02:44:48.210173 27113 sys_catalog.cc:564] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:48.211431 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de74ebcc05134eb5aa30f1b2a68165b6. Latest consensus state: current_term: 1 leader_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260806 02:44:48.211585 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:48.211984 27125 sys_catalog.cc:455] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de74ebcc05134eb5aa30f1b2a68165b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260806 02:44:48.213411 27125 sys_catalog.cc:458] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6 [sys.catalog]: This master's current role is: LEADER
W20260806 02:44:48.215417 27143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:44:48.215514 27143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:44:48.215590 27134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:48.216329 27134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:48.222283 27134 catalog_manager.cc:1347] Generated new cluster ID: 8f5af11cd5f14730a5e2cec4ba002606
I20260806 02:44:48.222358 27134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:48.233618 27134 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:48.233968 27134 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:48.242651 27134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de74ebcc05134eb5aa30f1b2a68165b6: Generated new TSK 0
I20260806 02:44:48.243522 27134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:48.247356 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27022
2026-08-06T02:44:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1649 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:48 dist-test-slave-tfpf krb5kdc[27151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:48 dist-test-slave-tfpf krb5kdc[27151](info): set up 2 sockets
Aug 06 02:44:48 dist-test-slave-tfpf krb5kdc[27151](info): commencing operation
krb5kdc: starting...
W20260806 02:44:49.511227 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:49.650475 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:49.765304 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:49.765626 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:49.765717 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:49.769733 27169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:49.769831 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:49.769873 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:49.769909 27169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:49.769940 27169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:49.775264 27169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:49.776404 27169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:49.777518 27169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:49.786168 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:49.786154 27177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:49.786156 27178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:49.786337 27169 server_base.cc:1034] running on GCE node
I20260806 02:44:49.787079 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:49.788247 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:49.789448 27169 hybrid_clock.cc:648] HybridClock initialized: now 1785984289789429 us; error 31 us; skew 500 ppm
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:49.796257 27169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:49.797236 27169 webserver.cc:460] Webserver started at http://127.0.0.1:44983/ using document root <none> and password file <none>
I20260806 02:44:49.797809 27169 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:49.797906 27169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:49.798156 27169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:49.800276 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "8c3be3dafb5c47d1b710a91d8dcaae37"
format_stamp: "Formatted at 2026-08-06 02:44:49 on dist-test-slave-tfpf"
server_key: "ff9c9ca8c203cd6b5c7e58fcbd8c129e"
server_key_iv: "390e495e22c6f3a52493763f9213c857"
server_key_version: "encryptionkey@0"
I20260806 02:44:49.800912 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "8c3be3dafb5c47d1b710a91d8dcaae37"
format_stamp: "Formatted at 2026-08-06 02:44:49 on dist-test-slave-tfpf"
server_key: "ff9c9ca8c203cd6b5c7e58fcbd8c129e"
server_key_iv: "390e495e22c6f3a52493763f9213c857"
server_key_version: "encryptionkey@0"
I20260806 02:44:49.804747 27169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:44:49.807417 27192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:49.808678 27169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:44:49.808830 27169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "8c3be3dafb5c47d1b710a91d8dcaae37"
format_stamp: "Formatted at 2026-08-06 02:44:49 on dist-test-slave-tfpf"
server_key: "ff9c9ca8c203cd6b5c7e58fcbd8c129e"
server_key_iv: "390e495e22c6f3a52493763f9213c857"
server_key_version: "encryptionkey@0"
I20260806 02:44:49.808951 27169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:49.827529 27169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:49.830940 27169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:49.831185 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:49.839740 27169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33211
I20260806 02:44:49.839777 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33211 every 8 connection(s)
I20260806 02:44:49.840812 27169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:49.843672 27261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:49.848562 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27169
I20260806 02:44:49.848711 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:49.849004 29436 external_mini_cluster.cc:1428] Setting key d5b6b682e829e741765472d697a638b4
I20260806 02:44:49.849529 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37: Bootstrap starting.
I20260806 02:44:49.851889 27261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:49.852836 27261 log.cc:824] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:49.854996 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37: No bootstrap required, opened a new log
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:44:49.859450 27261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } }
I20260806 02:44:49.859781 27261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:49.859880 27261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3be3dafb5c47d1b710a91d8dcaae37, State: Initialized, Role: FOLLOWER
I20260806 02:44:49.860375 27261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } }
I20260806 02:44:49.860543 27261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:49.860646 27261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:49.860764 27261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:49.861703 27261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } }
I20260806 02:44:49.862048 27261 leader_election.cc:302] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c3be3dafb5c47d1b710a91d8dcaae37; no voters:
I20260806 02:44:49.862401 27261 leader_election.cc:288] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:49.862466 27267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:49.862699 27267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [term 1 LEADER]: Becoming Leader. State: Replica: 8c3be3dafb5c47d1b710a91d8dcaae37, State: Running, Role: LEADER
I20260806 02:44:49.863065 27267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } }
I20260806 02:44:49.863719 27261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:49.864662 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } } }
I20260806 02:44:49.864813 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:49.865049 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c3be3dafb5c47d1b710a91d8dcaae37. Latest consensus state: current_term: 1 leader_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3be3dafb5c47d1b710a91d8dcaae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33211 } } }
I20260806 02:44:49.865173 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:49.865233 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:49.850457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:45931 (local address 127.0.0.1:33211)
0806 02:44:49.850888 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:49.850905 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:49.850940 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:44:49.851562 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:49.851578 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:49.851900 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:49.852238 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:49.853192 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:49.854197 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:49.855377 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:49.855917 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:49.856069 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:49.859177 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:49.859183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:49.859197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:49.859227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:49.861440 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:49.862012 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:49.862018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:49.862024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:49.862199 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:49.862613 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:49.862624 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:49.862626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:49.863181 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:49.863488 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:49.863788 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:44:49.864209 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260806 02:44:49.866117 27280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:49.868494 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:49.869310 27280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:49.875682 27280 catalog_manager.cc:1347] Generated new cluster ID: 645674a89e7c4274b9a24ed24b69fcb9
I20260806 02:44:49.875756 27280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:49.880540 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:49.870689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:53011 (local address 127.0.0.1:33211)
0806 02:44:49.870803 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:49.870807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:49.871682 (+ 875us) server_negotiation.cc:407] Connection header received
0806 02:44:49.871788 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:49.871791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:49.871846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:49.871947 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:49.872808 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:49.873550 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:49.874420 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:49.874689 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:49.874813 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:49.876664 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:49.876668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:49.876671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:49.876697 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:49.878512 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:49.879111 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:49.879115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:49.879117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:49.879208 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:49.879597 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:49.879603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:49.879607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:49.879823 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:49.879975 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:49.880271 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 02:44:49.880374 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:44:49.896189 27280 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:49.897245 27280 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:49.903586 27280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c3be3dafb5c47d1b710a91d8dcaae37: Generated new TSK 0
I20260806 02:44:49.904281 27280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:49.911759 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27169
W20260806 02:44:49.924306 27149 connection.cc:535] client connection to 127.0.0.1:33211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33211): Connection reset by peer (error 104)
2026-08-06T02:44:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1658 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27301](info): set up 2 sockets
Aug 06 02:44:49 dist-test-slave-tfpf krb5kdc[27301](info): commencing operation
krb5kdc: starting...
W20260806 02:44:51.168458 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:51 dist-test-slave-tfpf krb5kdc[27301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:51.312942 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:51.434327 27317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:51.434638 27317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:51.434732 27317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:51.438833 27317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:51.438916 27317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:51.438964 27317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:51.439003 27317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:51.439057 27317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:51.444464 27317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:51.445554 27317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:51.446568 27317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:51.455305 27327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:51.455353 27325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:51.455312 27324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:51.455399 27317 server_base.cc:1034] running on GCE node
I20260806 02:44:51.456100 27317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:51.457324 27317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:51.458526 27317 hybrid_clock.cc:648] HybridClock initialized: now 1785984291458510 us; error 28 us; skew 500 ppm
Aug 06 02:44:51 dist-test-slave-tfpf krb5kdc[27301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:51.461359 27317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:51.462285 27317 webserver.cc:460] Webserver started at http://127.0.0.1:34809/ using document root <none> and password file <none>
I20260806 02:44:51.462873 27317 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:51.462967 27317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:51.463220 27317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:51.465408 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "62a84a39ef204d36ab6c153b9decd6f9"
format_stamp: "Formatted at 2026-08-06 02:44:51 on dist-test-slave-tfpf"
server_key: "6cf3d32644cb33cbd2519238ad9c287e"
server_key_iv: "245bc285c8e06c115b29bc2944e531b6"
server_key_version: "encryptionkey@0"
I20260806 02:44:51.465978 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "62a84a39ef204d36ab6c153b9decd6f9"
format_stamp: "Formatted at 2026-08-06 02:44:51 on dist-test-slave-tfpf"
server_key: "6cf3d32644cb33cbd2519238ad9c287e"
server_key_iv: "245bc285c8e06c115b29bc2944e531b6"
server_key_version: "encryptionkey@0"
I20260806 02:44:51.469885 27317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:44:51.472414 27342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:51.473923 27317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:44:51.474108 27317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "62a84a39ef204d36ab6c153b9decd6f9"
format_stamp: "Formatted at 2026-08-06 02:44:51 on dist-test-slave-tfpf"
server_key: "6cf3d32644cb33cbd2519238ad9c287e"
server_key_iv: "245bc285c8e06c115b29bc2944e531b6"
server_key_version: "encryptionkey@0"
I20260806 02:44:51.474241 27317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:51.491815 27317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:51.495610 27317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:51.495846 27317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:51.503999 27317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34351
I20260806 02:44:51.504017 27415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34351 every 8 connection(s)
I20260806 02:44:51.505119 27317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:51.508206 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:51.511783 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27317
I20260806 02:44:51.511958 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:51.512238 29436 external_mini_cluster.cc:1428] Setting key 46d9f90c6ee119e1f87bb81287b60254
I20260806 02:44:51.514704 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9: Bootstrap starting.
I20260806 02:44:51.517426 27418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:51.518249 27418 log.cc:824] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:51.520612 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9: No bootstrap required, opened a new log
I20260806 02:44:51.522280 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:51.513893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:48101 (local address 127.0.0.1:34351)
0806 02:44:51.514336 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:51.514364 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:51.514402 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:44:51.515157 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:51.515177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:51.515548 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:51.515847 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:51.516283 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:51.517463 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:51.518545 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:51.519095 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:51.519303 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:51.520358 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:51.520364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:51.520379 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:51.520857 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:51.520918 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:51.520944 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:51.521150 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:51.521346 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:44:51.521501 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
I20260806 02:44:51.525321 27418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } }
I20260806 02:44:51.525656 27418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:51.525730 27418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a84a39ef204d36ab6c153b9decd6f9, State: Initialized, Role: FOLLOWER
I20260806 02:44:51.526077 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:51.526270 27418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } }
I20260806 02:44:51.526454 27418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:51.526522 27418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:51.526638 27418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:51.528090 27418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } }
I20260806 02:44:51.528475 27418 leader_election.cc:302] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62a84a39ef204d36ab6c153b9decd6f9; no voters:
I20260806 02:44:51.528936 27418 leader_election.cc:288] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:51.529374 27431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:51.529649 27431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [term 1 LEADER]: Becoming Leader. State: Replica: 62a84a39ef204d36ab6c153b9decd6f9, State: Running, Role: LEADER
I20260806 02:44:51.530077 27431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } }
I20260806 02:44:51.530402 27418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:51.531810 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62a84a39ef204d36ab6c153b9decd6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } } }
I20260806 02:44:51.531952 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:51.532315 27433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62a84a39ef204d36ab6c153b9decd6f9. Latest consensus state: current_term: 1 leader_uuid: "62a84a39ef204d36ab6c153b9decd6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a84a39ef204d36ab6c153b9decd6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34351 } } }
I20260806 02:44:51.532480 27433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:51.533234 27444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:51.534694 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:51.528711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:39133 (local address 127.0.0.1:34351)
0806 02:44:51.528868 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:51.528876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:51.529103 (+ 227us) server_negotiation.cc:407] Connection header received
0806 02:44:51.529333 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:51.529337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:51.529391 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:51.529504 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:51.529955 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:51.530778 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:51.532179 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:51.532594 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:51.532723 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:51.533937 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:51.533941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:51.533944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:51.533960 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:51.533990 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:51.534005 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:51.534134 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:51.534545 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 02:44:51.534591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:44:51.537338 27444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:51.542900 27444 catalog_manager.cc:1347] Generated new cluster ID: e28e66192e33488f95ea209777fd6477
I20260806 02:44:51.543045 27444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:51.569967 27444 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:51.571034 27444 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:51.576701 27444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62a84a39ef204d36ab6c153b9decd6f9: Generated new TSK 0
I20260806 02:44:51.577507 27444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:51.585330 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27317
W20260806 02:44:51.595636 27299 connection.cc:535] client connection to 127.0.0.1:34351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34351): Connection reset by peer (error 104)
2026-08-06T02:44:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1671 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:51 dist-test-slave-tfpf krb5kdc[27457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:51 dist-test-slave-tfpf krb5kdc[27457](info): set up 2 sockets
Aug 06 02:44:51 dist-test-slave-tfpf krb5kdc[27457](info): commencing operation
krb5kdc: starting...
W20260806 02:44:52.810700 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:52 dist-test-slave-tfpf krb5kdc[27457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:44:52.944041 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:53.092909 27472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:53.093323 27472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:53.093425 27472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:53.098022 27472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:53.098129 27472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:53.098176 27472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:53.098219 27472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:53.098268 27472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:53.104341 27472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:53.105630 27472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:53.106709 27472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:53.115513 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:53.115584 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:53.115568 27480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:53.115870 27472 server_base.cc:1034] running on GCE node
I20260806 02:44:53.116468 27472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:53.117659 27472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:53.118848 27472 hybrid_clock.cc:648] HybridClock initialized: now 1785984293118819 us; error 42 us; skew 500 ppm
Aug 06 02:44:53 dist-test-slave-tfpf krb5kdc[27457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:53.122088 27472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:44:53.123045 27472 webserver.cc:460] Webserver started at http://127.0.0.1:38303/ using document root <none> and password file <none>
I20260806 02:44:53.123683 27472 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:53.123759 27472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:53.123960 27472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:53.126240 27472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "cd1ce51f32224459904872097769abda"
format_stamp: "Formatted at 2026-08-06 02:44:53 on dist-test-slave-tfpf"
server_key: "09a8f660eeba1caf23bdf9fcb36457fd"
server_key_iv: "515b2af8924a61f010b8c4c4d2e6be6a"
server_key_version: "encryptionkey@0"
I20260806 02:44:53.126844 27472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "cd1ce51f32224459904872097769abda"
format_stamp: "Formatted at 2026-08-06 02:44:53 on dist-test-slave-tfpf"
server_key: "09a8f660eeba1caf23bdf9fcb36457fd"
server_key_iv: "515b2af8924a61f010b8c4c4d2e6be6a"
server_key_version: "encryptionkey@0"
I20260806 02:44:53.131078 27472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:53.133667 27492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:53.134949 27472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:53.135128 27472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "cd1ce51f32224459904872097769abda"
format_stamp: "Formatted at 2026-08-06 02:44:53 on dist-test-slave-tfpf"
server_key: "09a8f660eeba1caf23bdf9fcb36457fd"
server_key_iv: "515b2af8924a61f010b8c4c4d2e6be6a"
server_key_version: "encryptionkey@0"
I20260806 02:44:53.135270 27472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:53.140046 27472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:53.143319 27472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:53.143512 27472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:53.150913 27472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33531
I20260806 02:44:53.150939 27571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33531 every 8 connection(s)
I20260806 02:44:53.151876 27472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:53.152302 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27472
I20260806 02:44:53.152511 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:53.152848 29436 external_mini_cluster.cc:1428] Setting key 2382dc4ac49036850997d3d6994e7dd7
I20260806 02:44:53.155126 27572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:53.158861 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:53.154443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:59215 (local address 127.0.0.1:33531)
0806 02:44:53.154893 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:53.154911 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:53.154948 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:44:53.155684 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:53.155703 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:53.155992 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:53.156435 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:53.157277 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:53.157283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:53.157298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:53.157308 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:53.157359 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:53.157379 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:53.157533 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:53.157607 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:44:53.157774 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":148,"threads_started":1}
I20260806 02:44:53.161939 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:53.162227 27572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda: Bootstrap starting.
I20260806 02:44:53.164757 27572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:53.165829 27572 log.cc:824] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:53.166035 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:53.164098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.74:37933 (local address 127.0.0.1:33531)
0806 02:44:53.164223 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:53.164227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:53.164372 (+ 145us) server_negotiation.cc:407] Connection header received
0806 02:44:53.164475 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:53.164478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:53.164525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:53.164612 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:53.165491 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:53.165496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:53.165499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:53.165584 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:53.165648 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:53.165672 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:53.165781 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:53.165877 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:44:53.165920 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:44:53.168165 27572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda: No bootstrap required, opened a new log
I20260806 02:44:53.172087 27572 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260806 02:44:53.172329 27572 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:53.172367 27572 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd1ce51f32224459904872097769abda, State: Initialized, Role: FOLLOWER
I20260806 02:44:53.172803 27572 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260806 02:44:53.172946 27572 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:53.173000 27572 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:53.173143 27572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:53.174085 27572 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260806 02:44:53.174399 27572 leader_election.cc:302] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd1ce51f32224459904872097769abda; no voters:
I20260806 02:44:53.174672 27572 leader_election.cc:288] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:53.174790 27586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:53.175092 27586 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [term 1 LEADER]: Becoming Leader. State: Replica: cd1ce51f32224459904872097769abda, State: Running, Role: LEADER
I20260806 02:44:53.175447 27586 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260806 02:44:53.176069 27572 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:53.177296 27588 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd1ce51f32224459904872097769abda. Latest consensus state: current_term: 1 leader_uuid: "cd1ce51f32224459904872097769abda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260806 02:44:53.177443 27588 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:53.177758 27587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd1ce51f32224459904872097769abda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd1ce51f32224459904872097769abda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260806 02:44:53.177882 27597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:53.177944 27587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:53.181272 27597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:53.187500 27597 catalog_manager.cc:1347] Generated new cluster ID: f0786b27239b447982c9f12d7b892eb3
I20260806 02:44:53.187585 27597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:53.195008 27597 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:53.195312 27597 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:53.200950 27597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd1ce51f32224459904872097769abda: Generated new TSK 0
I20260806 02:44:53.201586 27597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:53.212483 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27472
2026-08-06T02:44:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:53 dist-test-slave-tfpf krb5kdc[27616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:53 dist-test-slave-tfpf krb5kdc[27616](info): set up 2 sockets
Aug 06 02:44:53 dist-test-slave-tfpf krb5kdc[27616](info): commencing operation
krb5kdc: starting...
W20260806 02:44:54.450877 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:54.583894 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43573
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:54.702596 27634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:54.702886 27634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:54.702930 27634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:54.706918 27634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:54.706982 27634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:54.707003 27634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:54.707026 27634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:54.707041 27634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:54.712340 27634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:43573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:54.713418 27634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:54.714401 27634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:54.723198 27647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:54.723198 27644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:54.723268 27645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:54.723563 27634 server_base.cc:1034] running on GCE node
I20260806 02:44:54.724136 27634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:54.725319 27634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:54.726511 27634 hybrid_clock.cc:648] HybridClock initialized: now 1785984294726493 us; error 29 us; skew 500 ppm
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984294, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:54.729820 27634 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:54.730873 27634 webserver.cc:460] Webserver started at http://127.28.191.62:44719/ using document root <none> and password file <none>
I20260806 02:44:54.731490 27634 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:54.731560 27634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:54.731812 27634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:54.734135 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "4d5228fd5d3f4332a4340407fbafd2da"
format_stamp: "Formatted at 2026-08-06 02:44:54 on dist-test-slave-tfpf"
server_key: "cf1409fd87b235a626ca284add87996c"
server_key_iv: "1951b79da172ff0729112a3f65ea6682"
server_key_version: "encryptionkey@0"
I20260806 02:44:54.734756 27634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "4d5228fd5d3f4332a4340407fbafd2da"
format_stamp: "Formatted at 2026-08-06 02:44:54 on dist-test-slave-tfpf"
server_key: "cf1409fd87b235a626ca284add87996c"
server_key_iv: "1951b79da172ff0729112a3f65ea6682"
server_key_version: "encryptionkey@0"
I20260806 02:44:54.738874 27634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:54.741503 27656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:54.742687 27634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:54.742833 27634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "4d5228fd5d3f4332a4340407fbafd2da"
format_stamp: "Formatted at 2026-08-06 02:44:54 on dist-test-slave-tfpf"
server_key: "cf1409fd87b235a626ca284add87996c"
server_key_iv: "1951b79da172ff0729112a3f65ea6682"
server_key_version: "encryptionkey@0"
I20260806 02:44:54.742944 27634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:54.784698 27634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:54.787948 27634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:54.788163 27634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:54.795596 27634 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:43573
I20260806 02:44:54.795667 27736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:43573 every 8 connection(s)
I20260806 02:44:54.796661 27634 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:54.799630 27737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:54.801959 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27634
I20260806 02:44:54.802115 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:54.802434 29436 external_mini_cluster.cc:1428] Setting key e53e23d7ad981f8c0ce00260f7adb346
I20260806 02:44:54.806231 27737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da: Bootstrap starting.
I20260806 02:44:54.808485 27737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:54.809360 27737 log.cc:824] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:54.811452 27737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da: No bootstrap required, opened a new log
I20260806 02:44:54.815660 27737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } }
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:54.815925 27737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:54.816004 27737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5228fd5d3f4332a4340407fbafd2da, State: Initialized, Role: FOLLOWER
I20260806 02:44:54.816493 27737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } }
I20260806 02:44:54.816651 27737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:54.816746 27737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:54.816851 27737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:54.818073 27737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } }
I20260806 02:44:54.818434 27737 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5228fd5d3f4332a4340407fbafd2da; no voters:
I20260806 02:44:54.818791 27737 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:54.818872 27744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:54.819070 27744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [term 1 LEADER]: Becoming Leader. State: Replica: 4d5228fd5d3f4332a4340407fbafd2da, State: Running, Role: LEADER
I20260806 02:44:54.819429 27744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } }
I20260806 02:44:54.820152 27737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:54.821296 27746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5228fd5d3f4332a4340407fbafd2da. Latest consensus state: current_term: 1 leader_uuid: "4d5228fd5d3f4332a4340407fbafd2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } } }
I20260806 02:44:54.821470 27746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:54.821918 27758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:54.822276 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:54.803940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49029 (local address 127.28.191.62:43573)
0806 02:44:54.804417 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:54.804433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:54.804471 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:44:54.805377 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:54.805399 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:54.805760 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:54.806104 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:54.806945 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:54.807947 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:54.808886 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:54.809457 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:54.809627 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:54.816625 (+ 6998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:54.816631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:54.816664 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:54.816702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:54.818606 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:54.819288 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:54.819293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:54.819299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:54.819435 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:54.819810 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:54.819813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:54.819815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:54.820315 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:54.820550 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:54.820941 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:44:54.821240 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260806 02:44:54.823139 27745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5228fd5d3f4332a4340407fbafd2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5228fd5d3f4332a4340407fbafd2da" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 43573 } } }
I20260806 02:44:54.823240 27745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:54.825852 27758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:54.826027 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:54.832826 27758 catalog_manager.cc:1347] Generated new cluster ID: 6129dbbe940b49a7b6261457ab95950f
I20260806 02:44:54.832896 27758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:54.837954 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:54.828147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51553 (local address 127.28.191.62:43573)
0806 02:44:54.828283 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:54.828290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:54.828596 (+ 306us) server_negotiation.cc:407] Connection header received
0806 02:44:54.828724 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:54.828727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:54.828774 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:54.828859 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:54.829751 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:54.830802 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:54.831904 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:54.832170 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:54.832276 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:54.834281 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:54.834284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:54.834287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:54.834312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:54.836019 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:54.836574 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:54.836577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:54.836579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:54.836667 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:54.837020 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:54.837023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:54.837025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:54.837260 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:54.837397 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:54.837701 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 02:44:54.837798 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:44:54.842100 27758 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:54.843138 27758 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:54.848771 27758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5228fd5d3f4332a4340407fbafd2da: Generated new TSK 0
I20260806 02:44:54.849509 27758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:54.857784 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27634
W20260806 02:44:54.873526 27614 connection.cc:535] client connection to 127.28.191.62:43573 recv error: Network error: failed to read from TLS socket (remote: 127.28.191.62:43573): Connection reset by peer (error 104)
2026-08-06T02:44:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1648 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27777](info): set up 2 sockets
Aug 06 02:44:54 dist-test-slave-tfpf krb5kdc[27777](info): commencing operation
krb5kdc: starting...
W20260806 02:44:56.114668 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:56 dist-test-slave-tfpf krb5kdc[27777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:56.245215 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35325
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:56.362545 27805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:56.362845 27805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:56.362927 27805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:56.366997 27805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:56.367075 27805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:56.367115 27805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:56.367168 27805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:56.367215 27805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:56.372408 27805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:56.373555 27805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:56.374610 27805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:56.383085 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:56.383071 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:56.383402 27805 server_base.cc:1034] running on GCE node
W20260806 02:44:56.383599 27814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:56.384155 27805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:56.385363 27805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:56.386523 27805 hybrid_clock.cc:648] HybridClock initialized: now 1785984296386491 us; error 50 us; skew 500 ppm
Aug 06 02:44:56 dist-test-slave-tfpf krb5kdc[27777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984296, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:56.389453 27805 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:56.390393 27805 webserver.cc:460] Webserver started at http://127.28.191.62:36689/ using document root <none> and password file <none>
I20260806 02:44:56.390920 27805 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:56.390980 27805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:56.391148 27805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:56.393258 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "b37d1dad15b14ff1b2963a8d314dab9e"
format_stamp: "Formatted at 2026-08-06 02:44:56 on dist-test-slave-tfpf"
server_key: "10ee385d8bcb6cd91ae371cdc485c168"
server_key_iv: "13acb3cb5aa915b8f98eac83217995ab"
server_key_version: "encryptionkey@0"
I20260806 02:44:56.393810 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "b37d1dad15b14ff1b2963a8d314dab9e"
format_stamp: "Formatted at 2026-08-06 02:44:56 on dist-test-slave-tfpf"
server_key: "10ee385d8bcb6cd91ae371cdc485c168"
server_key_iv: "13acb3cb5aa915b8f98eac83217995ab"
server_key_version: "encryptionkey@0"
I20260806 02:44:56.397681 27805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:44:56.400077 27825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:56.401392 27805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:56.401530 27805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "b37d1dad15b14ff1b2963a8d314dab9e"
format_stamp: "Formatted at 2026-08-06 02:44:56 on dist-test-slave-tfpf"
server_key: "10ee385d8bcb6cd91ae371cdc485c168"
server_key_iv: "13acb3cb5aa915b8f98eac83217995ab"
server_key_version: "encryptionkey@0"
I20260806 02:44:56.401641 27805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:56.413640 27805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:56.416401 27805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:56.416604 27805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:56.423753 27805 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35325
I20260806 02:44:56.423789 27892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35325 every 8 connection(s)
I20260806 02:44:56.424700 27805 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:56.427596 27893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:56.433558 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: Bootstrap starting.
I20260806 02:44:56.433923 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27805
I20260806 02:44:56.434062 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:56.434331 29436 external_mini_cluster.cc:1428] Setting key 3ac41277a1e146f330c95be7eeafeb42
I20260806 02:44:56.436373 27893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:56.437314 27893 log.cc:824] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:56.439244 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: No bootstrap required, opened a new log
I20260806 02:44:56.442720 27893 raft_consensus.cc:348] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } }
I20260806 02:44:56.443077 27893 raft_consensus.cc:374] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:56.443167 27893 raft_consensus.cc:729] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37d1dad15b14ff1b2963a8d314dab9e, State: Initialized, Role: FOLLOWER
I20260806 02:44:56.443648 27893 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } }
I20260806 02:44:56.443745 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:56.435761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47853 (local address 127.28.191.62:35325)
0806 02:44:56.436203 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:56.436228 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:56.436274 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:44:56.437216 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:56.437239 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:56.437686 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:56.438046 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:56.438462 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:56.439465 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:56.440359 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:56.440834 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:56.441021 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:56.441959 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:56.441964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:56.441976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:56.442355 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:56.442410 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:56.442433 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:56.442579 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:56.442764 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:44:56.442902 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260806 02:44:56.443811 27893 raft_consensus.cc:388] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:56.443892 27893 raft_consensus.cc:482] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:56.444034 27893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:56.445210 27893 raft_consensus.cc:504] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } }
I20260806 02:44:56.445554 27893 leader_election.cc:302] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b37d1dad15b14ff1b2963a8d314dab9e; no voters:
I20260806 02:44:56.445907 27893 leader_election.cc:288] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:56.446262 27900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:56.446578 27900 raft_consensus.cc:686] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [term 1 LEADER]: Becoming Leader. State: Replica: b37d1dad15b14ff1b2963a8d314dab9e, State: Running, Role: LEADER
I20260806 02:44:56.446986 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:56.446942 27900 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } }
I20260806 02:44:56.447211 27893 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:56.448653 27902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } } }
I20260806 02:44:56.448804 27902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:56.449082 27904 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37d1dad15b14ff1b2963a8d314dab9e. Latest consensus state: current_term: 1 leader_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37d1dad15b14ff1b2963a8d314dab9e" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35325 } } }
I20260806 02:44:56.449249 27904 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e [sys.catalog]: This master's current role is: LEADER
W20260806 02:44:56.452418 27930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:44:56.452517 27930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:44:56.452587 27927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:56.453516 27927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:56.454308 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:56.450193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36205 (local address 127.28.191.62:35325)
0806 02:44:56.450288 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:56.450293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:56.450529 (+ 236us) server_negotiation.cc:407] Connection header received
0806 02:44:56.450673 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:56.450676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:56.450727 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:56.450811 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:56.451103 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:56.451633 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:56.452448 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:56.452787 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:56.452893 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:56.453826 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:56.453830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:56.453832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:56.453845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:56.453875 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:56.453886 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:56.453978 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:56.454122 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 02:44:56.454157 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 02:44:56.459165 27927 catalog_manager.cc:1347] Generated new cluster ID: 8689dbd1ec444b6f8569be031dab5301
I20260806 02:44:56.459259 27927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:56.471587 27927 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:56.472615 27927 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:56.482129 27927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b37d1dad15b14ff1b2963a8d314dab9e: Generated new TSK 0
I20260806 02:44:56.482889 27927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:56.488860 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27805
W20260806 02:44:56.505395 27775 connection.cc:535] client connection to 127.28.191.62:35325 recv error: Network error: failed to read from TLS socket (remote: 127.28.191.62:35325): Connection reset by peer (error 104)
2026-08-06T02:44:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:56 dist-test-slave-tfpf krb5kdc[27939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:56 dist-test-slave-tfpf krb5kdc[27939](info): set up 2 sockets
Aug 06 02:44:56 dist-test-slave-tfpf krb5kdc[27939](info): commencing operation
krb5kdc: starting...
W20260806 02:44:57.764122 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:57 dist-test-slave-tfpf krb5kdc[27939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:57.894209 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42387
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:58.010637 27959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:58.010977 27959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:58.011061 27959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:58.015172 27959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:58.015268 27959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:58.015307 27959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:58.015359 27959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:58.015399 27959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:58.020812 27959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:42387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:58.022014 27959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:58.023130 27959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:58.031988 27966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:58.032193 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:58.032045 27965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:58.032354 27959 server_base.cc:1034] running on GCE node
I20260806 02:44:58.033053 27959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:58.034256 27959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:58.035445 27959 hybrid_clock.cc:648] HybridClock initialized: now 1785984298035433 us; error 28 us; skew 500 ppm
Aug 06 02:44:58 dist-test-slave-tfpf krb5kdc[27939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:58.038447 27959 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:58.039348 27959 webserver.cc:460] Webserver started at http://127.28.191.62:41163/ using document root <none> and password file <none>
I20260806 02:44:58.039925 27959 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:58.040019 27959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:58.040264 27959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:58.042491 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "fe7fb7b506b347129eca6cb0c461ded1"
format_stamp: "Formatted at 2026-08-06 02:44:58 on dist-test-slave-tfpf"
server_key: "fafec855d3af3bcf40c099ef4e83b38c"
server_key_iv: "72a1d96772c08dceeeb643cdfd3b065a"
server_key_version: "encryptionkey@0"
I20260806 02:44:58.043090 27959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "fe7fb7b506b347129eca6cb0c461ded1"
format_stamp: "Formatted at 2026-08-06 02:44:58 on dist-test-slave-tfpf"
server_key: "fafec855d3af3bcf40c099ef4e83b38c"
server_key_iv: "72a1d96772c08dceeeb643cdfd3b065a"
server_key_version: "encryptionkey@0"
I20260806 02:44:58.046999 27959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:58.049530 27982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:58.050865 27959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:44:58.051003 27959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "fe7fb7b506b347129eca6cb0c461ded1"
format_stamp: "Formatted at 2026-08-06 02:44:58 on dist-test-slave-tfpf"
server_key: "fafec855d3af3bcf40c099ef4e83b38c"
server_key_iv: "72a1d96772c08dceeeb643cdfd3b065a"
server_key_version: "encryptionkey@0"
I20260806 02:44:58.051117 27959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:58.055734 27959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:58.058869 27959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:58.059070 27959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:58.066541 27959 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:42387
I20260806 02:44:58.066571 28057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:42387 every 8 connection(s)
I20260806 02:44:58.067476 27959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:58.070226 28058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:58.071244 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 27959
I20260806 02:44:58.071372 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:58.071645 29436 external_mini_cluster.cc:1428] Setting key d0d4e27ff98511e56aeab3c564a999a6
I20260806 02:44:58.076508 28058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1: Bootstrap starting.
I20260806 02:44:58.076838 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:58.073038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51599 (local address 127.28.191.62:42387)
0806 02:44:58.073507 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:58.073521 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:58.073557 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:44:58.074203 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:58.074219 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:58.074484 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:58.074826 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:58.075642 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:58.075648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:58.075666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:58.075677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:58.075730 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:58.075749 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:58.075898 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:58.075969 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 02:44:58.076124 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260806 02:44:58.079100 28058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:58.079625 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:58.079932 28058 log.cc:824] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:58.082054 28058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1: No bootstrap required, opened a new log
I20260806 02:44:58.083391 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:58.081706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.28.191.62:42387)
0806 02:44:58.081832 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:58.081836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:58.082032 (+ 196us) server_negotiation.cc:407] Connection header received
0806 02:44:58.082192 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:58.082195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:58.082228 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:58.082301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:58.082990 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:58.082993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:58.082995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:44:58.083005 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:58.083028 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:44:58.083037 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:58.083104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:58.083231 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 02:44:58.083263 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:44:58.085987 28058 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } }
I20260806 02:44:58.086225 28058 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:58.086283 28058 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7fb7b506b347129eca6cb0c461ded1, State: Initialized, Role: FOLLOWER
I20260806 02:44:58.086714 28058 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } }
I20260806 02:44:58.086862 28058 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:58.086930 28058 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:58.087029 28058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:44:58.087937 28058 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } }
I20260806 02:44:58.088285 28058 leader_election.cc:302] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe7fb7b506b347129eca6cb0c461ded1; no voters:
I20260806 02:44:58.088621 28058 leader_election.cc:288] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:58.088798 28070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:58.089043 28070 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [term 1 LEADER]: Becoming Leader. State: Replica: fe7fb7b506b347129eca6cb0c461ded1, State: Running, Role: LEADER
I20260806 02:44:58.089399 28070 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } }
I20260806 02:44:58.089922 28058 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:58.091261 28074 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe7fb7b506b347129eca6cb0c461ded1. Latest consensus state: current_term: 1 leader_uuid: "fe7fb7b506b347129eca6cb0c461ded1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } } }
I20260806 02:44:58.091272 28071 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe7fb7b506b347129eca6cb0c461ded1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe7fb7b506b347129eca6cb0c461ded1" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 42387 } } }
I20260806 02:44:58.091404 28074 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:58.091404 28071 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:58.091773 28084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:58.094595 28084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:58.100212 28084 catalog_manager.cc:1347] Generated new cluster ID: 385f1f59347d4c6bb636a8134e169bb0
I20260806 02:44:58.100277 28084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:58.112463 28084 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:58.112807 28084 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:58.127513 28084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe7fb7b506b347129eca6cb0c461ded1: Generated new TSK 0
I20260806 02:44:58.128432 28084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:58.140060 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 27959
2026-08-06T02:44:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1648 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:58 dist-test-slave-tfpf krb5kdc[28104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:58 dist-test-slave-tfpf krb5kdc[28104](info): set up 2 sockets
Aug 06 02:44:58 dist-test-slave-tfpf krb5kdc[28104](info): commencing operation
krb5kdc: starting...
W20260806 02:44:59.364727 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:44:59.493624 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36085
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:36085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:44:59.616410 28128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:44:59.616765 28128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:44:59.616858 28128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:44:59.620886 28128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:44:59.620978 28128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:44:59.621018 28128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:44:59.621093 28128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:44:59.621141 28128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:44:59.626397 28128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:36085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:44:59.627542 28128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:44:59.628607 28128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:44:59.637240 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:59.637262 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:44:59.637251 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:44:59.637488 28128 server_base.cc:1034] running on GCE node
I20260806 02:44:59.638084 28128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:44:59.639195 28128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:44:59.640384 28128 hybrid_clock.cc:648] HybridClock initialized: now 1785984299640366 us; error 30 us; skew 500 ppm
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:44:59.643113 28128 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:44:59.644006 28128 webserver.cc:460] Webserver started at http://127.28.191.62:44251/ using document root <none> and password file <none>
I20260806 02:44:59.644579 28128 fs_manager.cc:359] Metadata directory not provided
I20260806 02:44:59.644690 28128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:44:59.644918 28128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:44:59.646978 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "7a3c279e75fc424ebc91c30e24285a04"
format_stamp: "Formatted at 2026-08-06 02:44:59 on dist-test-slave-tfpf"
server_key: "040a2b525b24093f7c9a75b6035217e0"
server_key_iv: "bfe794bcd72a05f2740bee4c1d11eae5"
server_key_version: "encryptionkey@0"
I20260806 02:44:59.647562 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "7a3c279e75fc424ebc91c30e24285a04"
format_stamp: "Formatted at 2026-08-06 02:44:59 on dist-test-slave-tfpf"
server_key: "040a2b525b24093f7c9a75b6035217e0"
server_key_iv: "bfe794bcd72a05f2740bee4c1d11eae5"
server_key_version: "encryptionkey@0"
I20260806 02:44:59.651445 28128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:44:59.653936 28151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:44:59.655198 28128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 02:44:59.655339 28128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "7a3c279e75fc424ebc91c30e24285a04"
format_stamp: "Formatted at 2026-08-06 02:44:59 on dist-test-slave-tfpf"
server_key: "040a2b525b24093f7c9a75b6035217e0"
server_key_iv: "bfe794bcd72a05f2740bee4c1d11eae5"
server_key_version: "encryptionkey@0"
I20260806 02:44:59.655448 28128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:44:59.667790 28128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:44:59.671171 28128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:44:59.671418 28128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:44:59.680979 28128 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:36085
I20260806 02:44:59.681027 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:36085 every 8 connection(s)
I20260806 02:44:59.682174 28128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:44:59.685284 28221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:44:59.690955 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04: Bootstrap starting.
I20260806 02:44:59.691177 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28128
I20260806 02:44:59.691334 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:44:59.691612 29436 external_mini_cluster.cc:1428] Setting key 2e200178710e231556b05f9c29783dca
I20260806 02:44:59.693625 28221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04: Neither blocks nor log segments found. Creating new log.
I20260806 02:44:59.694365 28221 log.cc:824] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04: Log is configured to *not* fsync() on all Append() calls
I20260806 02:44:59.696257 28221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04: No bootstrap required, opened a new log
I20260806 02:44:59.699867 28221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } }
I20260806 02:44:59.700187 28221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:44:59.700268 28221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3c279e75fc424ebc91c30e24285a04, State: Initialized, Role: FOLLOWER
I20260806 02:44:59.700738 28221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } }
I20260806 02:44:59.700897 28221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:44:59.700980 28221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:44:59.701107 28221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:44:59.702409 28221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } }
I20260806 02:44:59.702765 28221 leader_election.cc:302] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3c279e75fc424ebc91c30e24285a04; no voters:
I20260806 02:44:59.703104 28221 leader_election.cc:288] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:44:59.703197 28231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:44:59.703379 28231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [term 1 LEADER]: Becoming Leader. State: Replica: 7a3c279e75fc424ebc91c30e24285a04, State: Running, Role: LEADER
I20260806 02:44:59.703693 28231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } }
I20260806 02:44:59.704473 28221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:44:59.705510 28233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3c279e75fc424ebc91c30e24285a04. Latest consensus state: current_term: 1 leader_uuid: "7a3c279e75fc424ebc91c30e24285a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } } }
I20260806 02:44:59.705735 28233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:59.706151 28232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3c279e75fc424ebc91c30e24285a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3c279e75fc424ebc91c30e24285a04" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 36085 } } }
I20260806 02:44:59.706337 28232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04 [sys.catalog]: This master's current role is: LEADER
I20260806 02:44:59.706728 28240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:44:59.710116 28240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:44:59.710503 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:59.693003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51129 (local address 127.28.191.62:36085)
0806 02:44:59.693540 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:59.693560 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:59.693603 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:44:59.694370 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:59.694391 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:59.694821 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:59.695209 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:59.696245 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:59.697511 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:59.698529 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:59.699096 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:59.699306 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:59.702826 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:59.702833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:59.702875 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:59.702934 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:59.707105 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:59.707639 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:59.707646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:59.707652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:59.707772 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:59.708128 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:59.708134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:59.708136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:59.708683 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:59.708885 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:59.709263 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:44:59.709564 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":141,"threads_started":1}
I20260806 02:44:59.713847 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:44:59.716178 28240 catalog_manager.cc:1347] Generated new cluster ID: 93a51e7c827049a5b80aa42406c85550
I20260806 02:44:59.716245 28240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:44:59.724452 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:44:59.716595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53997 (local address 127.28.191.62:36085)
0806 02:44:59.716714 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 02:44:59.716718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:44:59.716741 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:44:59.716783 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:44:59.716786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:44:59.716821 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:44:59.716910 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:44:59.717704 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:59.718281 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:59.719170 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:44:59.719442 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:44:59.719578 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:44:59.721030 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:44:59.721035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:44:59.721037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:44:59.721099 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:44:59.722658 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:59.723115 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:59.723119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:59.723121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:59.723206 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:44:59.723492 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:44:59.723498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:44:59.723500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:44:59.723691 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:44:59.723832 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:44:59.724120 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:44:59.724248 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:44:59.786852 28240 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:44:59.788206 28240 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:44:59.794924 28240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a3c279e75fc424ebc91c30e24285a04: Generated new TSK 0
I20260806 02:44:59.795924 28240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:44:59.811046 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28128
2026-08-06T02:44:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1671 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28267](info): set up 2 sockets
Aug 06 02:44:59 dist-test-slave-tfpf krb5kdc[28267](info): commencing operation
krb5kdc: starting...
W20260806 02:45:01.052064 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:45:01 dist-test-slave-tfpf krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:45:01.183629 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35223
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:01.313293 28286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:01.313642 28286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:01.313745 28286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:01.317962 28286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:01.318102 28286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:01.318149 28286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:01.318209 28286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:01.318251 28286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:01.323807 28286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:01.324957 28286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:01.326071 28286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:01.335145 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:01.335144 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:01.335407 28286 server_base.cc:1034] running on GCE node
W20260806 02:45:01.335523 28297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:01.336194 28286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:01.337399 28286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:01.338603 28286 hybrid_clock.cc:648] HybridClock initialized: now 1785984301338575 us; error 52 us; skew 500 ppm
Aug 06 02:45:01 dist-test-slave-tfpf krb5kdc[28267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984301, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:01.341627 28286 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:45:01.342582 28286 webserver.cc:460] Webserver started at http://127.28.191.62:42041/ using document root <none> and password file <none>
I20260806 02:45:01.343243 28286 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:01.343314 28286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:01.343545 28286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:01.346115 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd"
format_stamp: "Formatted at 2026-08-06 02:45:01 on dist-test-slave-tfpf"
server_key: "6146e340b6a2e81b2745f76c293f4359"
server_key_iv: "91cbf06071d16d44f1f3f50ca180fc2b"
server_key_version: "encryptionkey@0"
I20260806 02:45:01.346714 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd"
format_stamp: "Formatted at 2026-08-06 02:45:01 on dist-test-slave-tfpf"
server_key: "6146e340b6a2e81b2745f76c293f4359"
server_key_iv: "91cbf06071d16d44f1f3f50ca180fc2b"
server_key_version: "encryptionkey@0"
I20260806 02:45:01.350919 28286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:45:01.353796 28305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:01.355271 28286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:01.355436 28286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd"
format_stamp: "Formatted at 2026-08-06 02:45:01 on dist-test-slave-tfpf"
server_key: "6146e340b6a2e81b2745f76c293f4359"
server_key_iv: "91cbf06071d16d44f1f3f50ca180fc2b"
server_key_version: "encryptionkey@0"
I20260806 02:45:01.355568 28286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:01.381018 28286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:01.384363 28286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:01.384583 28286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:01.391850 28286 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:35223
I20260806 02:45:01.391888 28374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:35223 every 8 connection(s)
I20260806 02:45:01.392896 28286 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:01.395831 28375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:01.402527 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28286
I20260806 02:45:01.402650 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:45:01.402808 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd: Bootstrap starting.
I20260806 02:45:01.402974 29436 external_mini_cluster.cc:1428] Setting key 4b6cc96a9c88c2310d6fdd4603156973
I20260806 02:45:01.405305 28375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:01.406039 28375 log.cc:824] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:01.407845 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd: No bootstrap required, opened a new log
I20260806 02:45:01.411439 28375 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } }
I20260806 02:45:01.411630 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:01.404340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58017 (local address 127.28.191.62:35223)
0806 02:45:01.404757 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:01.404773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:01.404808 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:45:01.405517 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:01.405538 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:01.405864 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:01.406192 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:01.406588 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:01.407518 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:01.408455 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:01.408929 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:01.409146 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:01.409976 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:01.409981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:01.409994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:01.410381 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:01.410433 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:01.410456 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:01.410558 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:01.410706 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:45:01.410907 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260806 02:45:01.411685 28375 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:01.411751 28375 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e8e2ca6fee4677b0c17d627ff81ddd, State: Initialized, Role: FOLLOWER
I20260806 02:45:01.412252 28375 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } }
I20260806 02:45:01.412412 28375 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:01.412475 28375 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:01.412571 28375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:01.413609 28375 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } }
I20260806 02:45:01.413928 28375 leader_election.cc:302] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2e8e2ca6fee4677b0c17d627ff81ddd; no voters:
I20260806 02:45:01.414278 28375 leader_election.cc:288] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:01.414348 28383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:01.414565 28383 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [term 1 LEADER]: Becoming Leader. State: Replica: e2e8e2ca6fee4677b0c17d627ff81ddd, State: Running, Role: LEADER
I20260806 02:45:01.414698 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:01.414924 28383 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } }
I20260806 02:45:01.415573 28375 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:01.416621 28385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } } }
I20260806 02:45:01.416788 28385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:01.417009 28384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e8e2ca6fee4677b0c17d627ff81ddd. Latest consensus state: current_term: 1 leader_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e8e2ca6fee4677b0c17d627ff81ddd" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 35223 } } }
I20260806 02:45:01.417153 28384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:01.417479 28403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:01.420503 28403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:01.423563 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:01.416965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35943 (local address 127.28.191.62:35223)
0806 02:45:01.417098 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:01.417103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:01.417416 (+ 313us) server_negotiation.cc:407] Connection header received
0806 02:45:01.417617 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:01.417621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:01.417666 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:01.417715 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:01.418127 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:01.418861 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:01.421846 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:01.422189 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:01.422298 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:01.423109 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:01.423113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:01.423116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:01.423129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:01.423167 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:01.423178 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:01.423255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:01.423374 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:45:01.423425 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260806 02:45:01.426039 28403 catalog_manager.cc:1347] Generated new cluster ID: e1ce8b90cc3243aa9b1b2957d38d4827
I20260806 02:45:01.426103 28403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:01.443367 28403 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:01.444713 28403 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:01.450632 28403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2e8e2ca6fee4677b0c17d627ff81ddd: Generated new TSK 0
I20260806 02:45:01.451355 28403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:01.457144 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28286
2026-08-06T02:45:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1645 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:01 dist-test-slave-tfpf krb5kdc[28421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:01 dist-test-slave-tfpf krb5kdc[28421](info): set up 2 sockets
Aug 06 02:45:01 dist-test-slave-tfpf krb5kdc[28421](info): commencing operation
krb5kdc: starting...
W20260806 02:45:02.728611 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:45:02 dist-test-slave-tfpf krb5kdc[28421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:45:02.859959 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40867
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:46257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:02.979871 28437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:02.980386 28437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:02.980516 28437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:02.984656 28437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:02.984752 28437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:02.984800 28437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:02.984835 28437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:02.984867 28437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:02.990398 28437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:46257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:02.991636 28437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:02.992775 28437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:03.002035 28445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:03.002082 28437 server_base.cc:1034] running on GCE node
W20260806 02:45:03.002043 28447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:03.002056 28444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:03.002908 28437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:03.004137 28437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:03.005342 28437 hybrid_clock.cc:648] HybridClock initialized: now 1785984303005312 us; error 43 us; skew 500 ppm
Aug 06 02:45:03 dist-test-slave-tfpf krb5kdc[28421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984303, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:03.013362 28437 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:45:03.014405 28437 webserver.cc:460] Webserver started at http://127.28.191.62:35203/ using document root <none> and password file <none>
I20260806 02:45:03.015029 28437 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:03.015131 28437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:03.015393 28437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:03.017640 28437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "de5995ff66ff460fae440f51374d7fba"
format_stamp: "Formatted at 2026-08-06 02:45:03 on dist-test-slave-tfpf"
server_key: "5877c5c57060e1bda82a4830c9e751be"
server_key_iv: "7196bed72c966408742ac2fb9116dda0"
server_key_version: "encryptionkey@0"
I20260806 02:45:03.018249 28437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "de5995ff66ff460fae440f51374d7fba"
format_stamp: "Formatted at 2026-08-06 02:45:03 on dist-test-slave-tfpf"
server_key: "5877c5c57060e1bda82a4830c9e751be"
server_key_iv: "7196bed72c966408742ac2fb9116dda0"
server_key_version: "encryptionkey@0"
I20260806 02:45:03.022253 28437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:03.024868 28458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:03.026278 28437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:03.026453 28437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "de5995ff66ff460fae440f51374d7fba"
format_stamp: "Formatted at 2026-08-06 02:45:03 on dist-test-slave-tfpf"
server_key: "5877c5c57060e1bda82a4830c9e751be"
server_key_iv: "7196bed72c966408742ac2fb9116dda0"
server_key_version: "encryptionkey@0"
I20260806 02:45:03.026585 28437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:03.031464 28437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:03.034519 28437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:03.034722 28437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:03.042142 28437 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:40867
I20260806 02:45:03.042171 28540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:40867 every 8 connection(s)
I20260806 02:45:03.043119 28437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:03.046075 28541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:03.047250 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28437
I20260806 02:45:03.047398 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:45:03.047704 29436 external_mini_cluster.cc:1428] Setting key 725defef5a4acb978200621ae3cd7b94
I20260806 02:45:03.052528 28541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba: Bootstrap starting.
I20260806 02:45:03.052917 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:03.049190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45123 (local address 127.28.191.62:40867)
0806 02:45:03.049622 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:03.049637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:03.049671 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:45:03.050299 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:03.050316 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:03.050594 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:03.050922 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:03.051748 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:03.051753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:03.051764 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:03.051775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:03.051828 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:03.051848 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:03.051998 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:03.052069 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 02:45:03.052221 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260806 02:45:03.055011 28541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:03.055871 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:03.055927 28541 log.cc:824] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:03.058045 28541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba: No bootstrap required, opened a new log
I20260806 02:45:03.059892 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:03.058170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48825 (local address 127.28.191.62:40867)
0806 02:45:03.058285 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:03.058290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:03.058515 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:45:03.058624 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:03.058627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:03.058656 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:03.058735 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:03.059461 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:03.059463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:03.059465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:03.059476 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:03.059500 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:03.059510 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:03.059579 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:03.059707 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 02:45:03.059740 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 02:45:03.062489 28541 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } }
I20260806 02:45:03.062748 28541 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:03.062808 28541 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5995ff66ff460fae440f51374d7fba, State: Initialized, Role: FOLLOWER
I20260806 02:45:03.063265 28541 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } }
I20260806 02:45:03.063421 28541 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:03.063495 28541 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:03.063608 28541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:03.064505 28541 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } }
I20260806 02:45:03.064862 28541 leader_election.cc:302] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5995ff66ff460fae440f51374d7fba; no voters:
I20260806 02:45:03.065255 28541 leader_election.cc:288] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:03.065325 28558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:03.065569 28558 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [term 1 LEADER]: Becoming Leader. State: Replica: de5995ff66ff460fae440f51374d7fba, State: Running, Role: LEADER
I20260806 02:45:03.065924 28558 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } }
I20260806 02:45:03.066637 28541 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:03.067762 28559 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5995ff66ff460fae440f51374d7fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } } }
I20260806 02:45:03.067795 28560 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5995ff66ff460fae440f51374d7fba. Latest consensus state: current_term: 1 leader_uuid: "de5995ff66ff460fae440f51374d7fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5995ff66ff460fae440f51374d7fba" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 40867 } } }
I20260806 02:45:03.067894 28559 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:03.067907 28560 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:03.068542 28568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:03.071545 28568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:03.077546 28568 catalog_manager.cc:1347] Generated new cluster ID: 7fc200546c1e47f29c7d1826f90fcb12
I20260806 02:45:03.077632 28568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:03.092581 28568 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:03.093031 28568 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:03.104216 28568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5995ff66ff460fae440f51374d7fba: Generated new TSK 0
I20260806 02:45:03.105346 28568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:03.115767 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28437
2026-08-06T02:45:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1665 ms)
[----------] 18 tests from AuthTokenIssuingTest (29512 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:03 dist-test-slave-tfpf krb5kdc[28585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:03 dist-test-slave-tfpf krb5kdc[28585](info): set up 2 sockets
Aug 06 02:45:03 dist-test-slave-tfpf krb5kdc[28585](info): commencing operation
krb5kdc: starting...
W20260806 02:45:04.382189 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:45:04.513773 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46397
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:04.630174 28599 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:04.630566 28599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:04.630637 28599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:04.630716 28599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:04.634752 28599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:04.634845 28599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:04.634882 28599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:04.634922 28599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:04.634958 28599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:04.640300 28599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:04.641472 28599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:04.642524 28599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:04.651289 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:04.651381 28607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:04.651533 28599 server_base.cc:1034] running on GCE node
W20260806 02:45:04.651793 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:04.652351 28599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:04.653556 28599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:04.654765 28599 hybrid_clock.cc:648] HybridClock initialized: now 1785984304654728 us; error 49 us; skew 500 ppm
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:04.657642 28599 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:45:04.658579 28599 webserver.cc:460] Webserver started at http://127.28.191.62:37697/ using document root <none> and password file <none>
I20260806 02:45:04.659173 28599 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:04.659265 28599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:04.659509 28599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:04.662019 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "91d20a8f1e5e456388822ddf43361515"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "09af427cc10ee5841ae3f6b0ffed24cd"
server_key_iv: "74e1397e3c1081c55f1097bd4c01f1a7"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.662674 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "91d20a8f1e5e456388822ddf43361515"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "09af427cc10ee5841ae3f6b0ffed24cd"
server_key_iv: "74e1397e3c1081c55f1097bd4c01f1a7"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.666644 28599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:04.669210 28621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:04.670478 28599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:04.670641 28599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "91d20a8f1e5e456388822ddf43361515"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "09af427cc10ee5841ae3f6b0ffed24cd"
server_key_iv: "74e1397e3c1081c55f1097bd4c01f1a7"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.670766 28599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:04.681377 28599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:04.684430 28599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:04.684597 28599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:04.691798 28599 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:46397
I20260806 02:45:04.692099 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:46397 every 8 connection(s)
I20260806 02:45:04.692811 28599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:04.695829 28708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:04.699800 28708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:04.701689 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28599
I20260806 02:45:04.701828 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:45:04.702112 29436 external_mini_cluster.cc:1428] Setting key 23856856eb24cfae30c9dc9ad5c70ee7
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.61@KRBTEST.COM" created.
I20260806 02:45:04.718196 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.702421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.28.191.62:46397)
0806 02:45:04.702915 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:04.702936 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:04.703255 (+ 319us) server_negotiation.cc:407] Connection header received
0806 02:45:04.704652 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:04.704672 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:04.704925 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:04.705956 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:04.706718 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.708187 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.709516 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.709947 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.710111 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.713219 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:04.713228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:04.713247 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:04.713277 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:04.714940 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.715466 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.715477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.715478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.715583 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.716008 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.716011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.716013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.716538 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:04.716707 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:04.717173 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 02:45:04.717441 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":330,"thread_start_us":157,"threads_started":1}
I20260806 02:45:04.718201 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.702422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46397 (local address 127.0.0.1:54036)
0806 02:45:04.703084 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.703126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:04.704244 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:04.705178 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:04.705189 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:04.705724 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:04.706509 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.706532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.708327 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.708335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.709345 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.709363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.710145 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.710153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.710372 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.711018 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:04.711041 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:04.712994 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:04.715077 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.715088 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.715102 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.715339 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.715702 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.715708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.715710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.715854 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.716685 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:04.716691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:04.716837 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 02:45:04.716983 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:45:04.717336 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"spinlock_wait_cycles":13696,"thread_start_us":158,"threads_started":1}
I20260806 02:45:04.720777 28708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:04.721697 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.721333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46736)
0806 02:45:04.721513 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.721612 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 02:45:04.721935 28622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
I20260806 02:45:04.722913 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.722610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46738)
0806 02:45:04.722758 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.722825 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:45:04.723160 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:34699: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
I20260806 02:45:04.732728 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:34699
--webserver_interface=127.28.191.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:04.772472 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } attempt: 1
I20260806 02:45:04.773484 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.773174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46748)
0806 02:45:04.773337 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.773404 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:45:04.774245 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.774031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46756)
0806 02:45:04.774138 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.774190 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:45:04.774376 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:34699: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
I20260806 02:45:04.842561 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } attempt: 2
I20260806 02:45:04.843564 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.843219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46758)
0806 02:45:04.843414 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.843482 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:45:04.844380 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.844133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46760)
0806 02:45:04.844259 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.844317 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:45:04.844527 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:34699: Network error: Client connection negotiation failed: client connection to 127.28.191.61:34699: connect: Connection refused (error 111)
W20260806 02:45:04.860185 28716 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:04.860551 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:04.860647 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:04.860749 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:04.865283 28716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:04.865375 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:04.865418 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:04.865460 28716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:04.865501 28716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:04.871366 28716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:34699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.191.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:04.872622 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:04.873842 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:04.883688 28726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:04.883678 28724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:04.883736 28723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:04.883967 28716 server_base.cc:1034] running on GCE node
I20260806 02:45:04.884711 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:04.885977 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:04.887146 28716 hybrid_clock.cc:648] HybridClock initialized: now 1785984304887129 us; error 30 us; skew 500 ppm
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:04.894364 28716 init.cc:377] Logged in from keytab as kudu/127.28.191.61@KRBTEST.COM (short username kudu)
I20260806 02:45:04.895388 28716 webserver.cc:460] Webserver started at http://127.28.191.61:34019/ using document root <none> and password file <none>
I20260806 02:45:04.895944 28716 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:04.896006 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:04.896183 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:04.898406 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data/instance:
uuid: "0e58f01be8cd41d9a3a5844e8a4d0163"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "d95304b699833cf76ee2659f1889d913"
server_key_iv: "c11600345e74a40935059a23a5aa797b"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.898953 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal/instance:
uuid: "0e58f01be8cd41d9a3a5844e8a4d0163"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "d95304b699833cf76ee2659f1889d913"
server_key_iv: "c11600345e74a40935059a23a5aa797b"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.903014 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:04.905699 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:04.907030 28716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:04.907184 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "0e58f01be8cd41d9a3a5844e8a4d0163"
format_stamp: "Formatted at 2026-08-06 02:45:04 on dist-test-slave-tfpf"
server_key: "d95304b699833cf76ee2659f1889d913"
server_key_iv: "c11600345e74a40935059a23a5aa797b"
server_key_version: "encryptionkey@0"
I20260806 02:45:04.907307 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:04.921868 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:04.925012 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:04.925288 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:04.933035 28716 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.61:34699
I20260806 02:45:04.933385 28811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.61:34699 every 8 connection(s)
I20260806 02:45:04.934104 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
I20260806 02:45:04.935717 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } attempt: 3
I20260806 02:45:04.937577 28812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:04.941035 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28716
I20260806 02:45:04.941238 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-1/wal/instance
I20260806 02:45:04.941644 29436 external_mini_cluster.cc:1428] Setting key f3792e9cb3a916dd44c84fb532a3f339
I20260806 02:45:04.942384 28812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:04.951893 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.936387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46766)
0806 02:45:04.936569 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.936598 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:04.936721 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:04.939268 (+ 2547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:04.939272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:04.939510 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:04.939689 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.939698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.941394 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.941399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.942294 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.942308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.943400 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.943405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.943545 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.944553 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:04.944571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:04.946398 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:04.949640 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.949646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.949648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.949878 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.950339 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.950346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.950350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.950481 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.951288 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:04.951292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:04.951411 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 02:45:04.951542 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:45:04.951708 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:45:04.953648 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.937260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.28.191.61:34699)
0806 02:45:04.937816 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:04.937833 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:04.937871 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:45:04.938569 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:04.938585 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:04.938943 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:04.939246 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:04.939855 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.941188 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.942453 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.943243 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.943452 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.946625 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:04.946637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:04.946655 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:04.946697 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:04.949506 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.950009 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.950018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.950026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.950188 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.950608 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.950612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.950614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.951153 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:04.951307 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:04.951824 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 02:45:04.952445 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":157,"threads_started":1}
WARNING: no policy specified for kudu/127.28.191.60@KRBTEST.COM; defaulting to no policy
I20260806 02:45:04.955963 28708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:04.956394 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.944460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46397 (local address 127.0.0.1:54038)
0806 02:45:04.945015 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.945104 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:04.945329 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:04.945742 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:04.945753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:04.946308 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:04.946539 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.946557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.947614 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.947616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.948455 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.948466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.949026 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.949031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.949217 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.950105 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:04.950120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:04.952133 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:04.954429 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.954438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.954443 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.954684 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.955042 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.955049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.955054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.955162 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.955656 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:04.955663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:04.955787 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 02:45:04.955967 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:45:04.956178 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":180,"threads_started":1}
I20260806 02:45:04.957152 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.956717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52496)
0806 02:45:04.956941 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.957013 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:45:04.957343 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.944545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.28.191.62:46397)
0806 02:45:04.944718 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:04.944723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:04.945203 (+ 480us) server_negotiation.cc:407] Connection header received
0806 02:45:04.945462 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:04.945467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:04.945529 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:04.945646 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:04.946692 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.947455 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.948592 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.948865 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.949019 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.952350 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:04.952354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:04.952357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:04.952387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:04.954306 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.954832 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.954836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.954838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.954925 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.955281 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.955285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.955287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.955512 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:04.955657 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:04.956923 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0806 02:45:04.957109 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.60@KRBTEST.COM" created.
I20260806 02:45:04.958081 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.957711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52510)
0806 02:45:04.957943 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.958002 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260806 02:45:04.958205 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
I20260806 02:45:04.958396 28812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:04 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:04.969995 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.958838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46778)
0806 02:45:04.958973 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.958988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:04.959082 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:04.959430 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:04.959433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:04.959580 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:04.959680 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.959685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.960649 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.960655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.961680 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:04.961694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.962417 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.962420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:04.962517 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.963036 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:04.963048 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:04.964707 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:04.967353 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.967360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.967365 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.967816 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.968286 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:04.968290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:04.968292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:04.968475 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:04.969233 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:04.969238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:04.969365 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 02:45:04.969526 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 02:45:04.969710 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":5}
I20260806 02:45:04.970250 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.959019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.28.191.61:34699)
0806 02:45:04.959177 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:04.959184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:04.959201 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:45:04.959247 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:04.959253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:04.959311 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:04.959407 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:04.959797 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.960520 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.961814 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:04.962288 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:04.962414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:04.964855 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:04.964858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:04.964861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:04.964883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:04.967196 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.968010 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.968015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.968017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.968136 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:04.968659 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:04.968665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:04.968669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:04.969023 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:04.969182 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:04.969643 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 02:45:04.969887 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:45:04.971309 28812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:04.972200 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.971865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52514)
0806 02:45:04.972020 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.972133 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 02:45:04.973045 28740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
I20260806 02:45:04.975944 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.975649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52520)
0806 02:45:04.975785 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.975861 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 02:45:04.976229 28812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
I20260806 02:45:04.980718 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:45379
--webserver_interface=127.28.191.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:04.989388 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 1
I20260806 02:45:04.990391 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.990066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52522)
0806 02:45:04.990239 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.990305 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:45:04.991235 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:04.990977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52538)
0806 02:45:04.991105 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:04.991168 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:45:04.991391 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
I20260806 02:45:05.025575 28812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 1
I20260806 02:45:05.026669 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.026252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52544)
0806 02:45:05.026455 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.026549 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:45:05.027576 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.027273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52560)
0806 02:45:05.027420 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.027491 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:45:05.027745 28812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
I20260806 02:45:05.067061 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 2
I20260806 02:45:05.068192 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.067774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52574)
0806 02:45:05.067971 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.068073 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 02:45:05.069242 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.068872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52586)
0806 02:45:05.069014 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.069132 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:45:05.069566 28708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
I20260806 02:45:05.095917 28812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 2
I20260806 02:45:05.096913 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.096547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52594)
0806 02:45:05.096747 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.096821 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:45:05.097872 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.097562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52604)
0806 02:45:05.097715 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.097785 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:45:05.098024 28812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:45379: Network error: Client connection negotiation failed: client connection to 127.28.191.60:45379: connect: Connection refused (error 111)
W20260806 02:45:05.103758 28830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:05.104072 28830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:05.104136 28830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:05.104216 28830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:05.109038 28830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:05.109170 28830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:05.109217 28830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:05.109252 28830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:05.109287 28830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:05.114822 28830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:46397,127.28.191.61:34699,127.28.191.60:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.191.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:05.115931 28830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:05.116986 28830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:05.125854 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:05.125901 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:05.125852 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:05.126260 28830 server_base.cc:1034] running on GCE node
I20260806 02:45:05.126843 28830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:05.127980 28830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:05.129166 28830 hybrid_clock.cc:648] HybridClock initialized: now 1785984305129134 us; error 56 us; skew 500 ppm
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:05.132133 28830 init.cc:377] Logged in from keytab as kudu/127.28.191.60@KRBTEST.COM (short username kudu)
I20260806 02:45:05.133199 28830 webserver.cc:460] Webserver started at http://127.28.191.60:40371/ using document root <none> and password file <none>
I20260806 02:45:05.133805 28830 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:05.133868 28830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:05.134092 28830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:05.136143 28830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data/instance:
uuid: "f6c22f121125462dadf753d327dbbdc6"
format_stamp: "Formatted at 2026-08-06 02:45:05 on dist-test-slave-tfpf"
server_key: "e108cb56edf4d384be4d9e5fb8555506"
server_key_iv: "7ee9ab58b1721cf987a142f0bf589644"
server_key_version: "encryptionkey@0"
I20260806 02:45:05.136715 28830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal/instance:
uuid: "f6c22f121125462dadf753d327dbbdc6"
format_stamp: "Formatted at 2026-08-06 02:45:05 on dist-test-slave-tfpf"
server_key: "e108cb56edf4d384be4d9e5fb8555506"
server_key_iv: "7ee9ab58b1721cf987a142f0bf589644"
server_key_version: "encryptionkey@0"
I20260806 02:45:05.140547 28830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:05.143119 28848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:05.144520 28830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:05.144695 28830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "f6c22f121125462dadf753d327dbbdc6"
format_stamp: "Formatted at 2026-08-06 02:45:05 on dist-test-slave-tfpf"
server_key: "e108cb56edf4d384be4d9e5fb8555506"
server_key_iv: "7ee9ab58b1721cf987a142f0bf589644"
server_key_version: "encryptionkey@0"
I20260806 02:45:05.144831 28830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:05.155465 28830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:05.158804 28830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:05.159009 28830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:05.166797 28830 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.60:45379
I20260806 02:45:05.167147 28922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.60:45379 every 8 connection(s)
I20260806 02:45:05.167834 28830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
I20260806 02:45:05.168766 28708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 3
I20260806 02:45:05.168988 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 28830
I20260806 02:45:05.169242 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984016215184-29436-0/minicluster-data/master-2/wal/instance
I20260806 02:45:05.169570 29436 external_mini_cluster.cc:1428] Setting key cb22e17cc7def9ae9467b475927f7f2c
I20260806 02:45:05.171844 28923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:05.178228 28923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:45:05.186846 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.169322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52616)
0806 02:45:05.169846 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.169863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:05.169975 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:05.173298 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:05.173302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:05.173538 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:05.173698 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.173708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.176468 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.176472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.177552 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.177564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.178608 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.178612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.178704 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.179672 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:05.179690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:05.181898 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:05.184688 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.184693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.184697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.185016 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.185410 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.185416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.185421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.185526 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.186115 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:05.186121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:05.186250 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 02:45:05.186356 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:05.186572 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418}
I20260806 02:45:05.188776 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.170772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.28.191.60:45379)
0806 02:45:05.171243 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.171525 (+ 282us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.171577 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:45:05.172404 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.172431 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.172906 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.174069 (+ 1163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.174576 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.176304 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.177687 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.178457 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.182109 (+ 3652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.182284 (+ 175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.182289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.182294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.182319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.184537 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.185193 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.185199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.185204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.185298 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.185658 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.185663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.185668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.185960 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.187198 (+ 1238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.187394 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.187776 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":563,"server-negotiator.queue_time_us":323,"spinlock_wait_cycles":43648,"thread_start_us":158,"threads_started":1}
I20260806 02:45:05.188786 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.171323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.28.191.60:45379)
0806 02:45:05.171605 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.171615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.171626 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:45:05.172404 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.172431 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.172906 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.174209 (+ 1303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.174582 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.176304 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.177540 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.178457 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.179021 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.182161 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.182170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.182185 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.182224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.184391 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.184948 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.184954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.184960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.185136 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.185444 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.185447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.185450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.185957 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.187205 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.187393 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.187724 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":208,"server-negotiator.queue_time_us":212,"spinlock_wait_cycles":15744,"thread_start_us":118,"threads_started":1}
I20260806 02:45:05.189179 28812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } attempt: 3
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:05.194893 28708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: Bootstrap starting.
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:05.199553 28708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:05.200978 28708 log.cc:824] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:05.201534 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.181550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.28.191.62:46397)
0806 02:45:05.181689 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.181693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.181706 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:05.181749 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.181755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.181810 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.181892 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.184651 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.185444 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.186701 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.187026 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.187102 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.192089 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.192096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.192100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.192127 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.194928 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.199693 (+ 4765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.199698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.199700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.199814 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.200187 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.200192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.200195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.200460 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.200567 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.201035 (+ 468us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.201384 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:45:05.201566 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.180601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:46397 (local address 127.0.0.1:54048)
0806 02:45:05.181050 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.181128 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:05.181247 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:05.182512 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:05.182522 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:05.184335 (+ 1813us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:05.184495 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.184506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.185585 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.185588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.186544 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.186569 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.187350 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.187353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.187469 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.190265 (+ 2796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:05.190279 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:05.191933 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:05.196887 (+ 4954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.196897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.196903 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.199551 (+ 2648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.199952 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.199956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.199962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.200064 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.200645 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:05.200652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:05.200781 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 02:45:05.201158 (+ 377us) client_negotiation.cc:240] Negotiation successful
0806 02:45:05.201366 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":129,"threads_started":1}
I20260806 02:45:05.203442 28923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:05.203696 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.189728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52620)
0806 02:45:05.189960 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.189977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:05.190085 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:05.192588 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:05.192591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:05.192784 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:05.192902 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.192913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.195056 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.195059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.195845 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.195855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.196527 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.196531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.196638 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.197174 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:05.197194 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:05.198638 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:05.202045 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.202048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.202050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.202323 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.202617 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.202622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.202624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.202720 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.203182 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:05.203185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:05.203286 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 02:45:05.203387 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:45:05.203522 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260806 02:45:05.204154 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.190005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.28.191.60:45379)
0806 02:45:05.192317 (+ 2312us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.192322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.192335 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:05.192374 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.192379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.192426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.192549 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.193422 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.194161 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.195965 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.196296 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.196393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.198796 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.198800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.198803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.198832 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.201928 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.202440 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.202443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.202445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.202524 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.202834 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.202841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.202846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.203047 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.203134 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.203887 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.203997 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2244,"thread_start_us":126,"threads_started":1}
I20260806 02:45:05.204753 28708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: No bootstrap required, opened a new log
I20260806 02:45:05.207962 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.191692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.28.191.62:46397)
0806 02:45:05.192785 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.192789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.192801 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:05.192834 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.192837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.192875 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.193739 (+ 864us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.193845 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.194592 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.195730 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.196072 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.196495 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.200684 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.200688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.200691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.200713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.205597 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.206096 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.206099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.206101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.206177 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.206504 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.206507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.206509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.206735 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.207561 (+ 826us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.207639 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.207766 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":87,"threads_started":1}
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:05.213598 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.204165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.28.191.61:34699)
0806 02:45:05.204295 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.204299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.204725 (+ 426us) server_negotiation.cc:407] Connection header received
0806 02:45:05.204837 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.204840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.204882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.204971 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.205487 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.206233 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.207309 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.207643 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.207724 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.210029 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.210032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.210035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.210065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.211864 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.212284 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.212288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.212290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.212370 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.212666 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.212670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.212671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.212840 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.212946 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.213326 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.213440 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:45:05.213848 28812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: Bootstrap starting.
I20260806 02:45:05.213858 28708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.214120 28708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d20a8f1e5e456388822ddf43361515, State: Initialized, Role: FOLLOWER
I20260806 02:45:05.214494 28708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.215896 28708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:05.217585 28812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:05.219062 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.203914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:34699 (local address 127.0.0.1:46780)
0806 02:45:05.204654 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.204670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:05.204756 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:05.205016 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:05.205020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:05.205215 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:05.205338 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.205347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.206363 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.206366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.207173 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.207186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.207872 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.207875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.207975 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.208439 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:05.208455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:05.209922 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:05.211972 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.211976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.211978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.212179 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.212473 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.212476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.212478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.212575 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.212937 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:05.212940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:05.213036 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 02:45:05.218719 (+ 5683us) client_negotiation.cc:240] Negotiation successful
0806 02:45:05.218880 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658}
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:05.219973 28923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:05.221801 28936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.221936 28936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.224112 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.209382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.28.191.61:34699)
0806 02:45:05.213743 (+ 4361us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.213746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.213759 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:05.213793 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.213795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.213832 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.213910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.214795 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.215449 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.216477 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.216873 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.216982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.219880 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.219884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.219887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.219912 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.221860 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.222381 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.222385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.222386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.222469 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.222778 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.222782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.222783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.222983 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.223156 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.223856 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.223962 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4306,"thread_start_us":135,"threads_started":1}
W20260806 02:45:05.225478 28953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:05.225548 28953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:05.221822 28599 master_runner.cc:186] Error getting master registration for 127.28.191.61:34699: OK, instance_id { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" instance_seqno: 1785984304930759 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:45:05.226969 28599 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:45:05.227624 28812 log.cc:824] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:05.228672 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:05.230363 28666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91d20a8f1e5e456388822ddf43361515"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54062
I20260806 02:45:05.230691 28666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:45:05.230803 28666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:45:05 dist-test-slave-tfpf krb5kdc[28585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:45:05.232193 28666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.233242 28666 leader_election.cc:288] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [CANDIDATE]: Term 1 election: Requested vote from peers 0e58f01be8cd41d9a3a5844e8a4d0163 (127.28.191.61:34699), f6c22f121125462dadf753d327dbbdc6 (127.28.191.60:45379)
I20260806 02:45:05.233865 28812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: No bootstrap required, opened a new log
I20260806 02:45:05.239428 28812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.239702 28812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e58f01be8cd41d9a3a5844e8a4d0163, State: Initialized, Role: FOLLOWER
I20260806 02:45:05.240109 28812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.242208 28812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:05.242372 28778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91d20a8f1e5e456388822ddf43361515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163"
I20260806 02:45:05.242622 28778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:05.244014 28778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d20a8f1e5e456388822ddf43361515 in term 1.
I20260806 02:45:05.244451 28622 leader_election.cc:302] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e58f01be8cd41d9a3a5844e8a4d0163, 91d20a8f1e5e456388822ddf43361515; no voters:
I20260806 02:45:05.244776 28936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:05.244982 28936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [term 1 LEADER]: Becoming Leader. State: Replica: 91d20a8f1e5e456388822ddf43361515, State: Running, Role: LEADER
I20260806 02:45:05.245368 28936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.247165 28936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d20a8f1e5e456388822ddf43361515. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.247285 28936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:05.251307 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.220650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.28.191.60:45379)
0806 02:45:05.220869 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.220872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.224460 (+ 3588us) server_negotiation.cc:407] Connection header received
0806 02:45:05.224613 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.224615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.224651 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.224693 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.225205 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.225923 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.226932 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.227229 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.227304 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.236402 (+ 9098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.236407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.236410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.236437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.238349 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.241533 (+ 3184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.241538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.241540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.241643 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.242157 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.242161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.242163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.250279 (+ 8116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.250380 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.250873 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.251064 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260806 02:45:05.251953 28961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.252097 28961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.252411 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.220438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:45379 (local address 127.0.0.1:52634)
0806 02:45:05.221026 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:05.224409 (+ 3383us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:05.224514 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:05.224806 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:05.224809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:05.224982 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:05.225093 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.225101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.226038 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.226041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.226806 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:05.226817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.227438 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.227442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:05.227529 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.228021 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:05.228094 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:05.231593 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:05.241125 (+ 9532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.241129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.241131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.241411 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.241783 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:05.241786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:05.241789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:05.241983 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:05.250437 (+ 8454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:05.250440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:05.250602 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 02:45:05.251754 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0806 02:45:05.251975 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515}
I20260806 02:45:05.256498 28923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: Bootstrap starting.
I20260806 02:45:05.252935 28716 master_runner.cc:186] Error getting master registration for 127.28.191.60:45379: OK, instance_id { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" instance_seqno: 1785984305164668 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:45:05.257076 28716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:45:05.259749 28923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:05.260751 28923 log.cc:824] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:05.264264 28923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: No bootstrap required, opened a new log
I20260806 02:45:05.272174 28989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:05.273500 28989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:05.274514 28923 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.275509 28923 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c22f121125462dadf753d327dbbdc6, State: Initialized, Role: FOLLOWER
I20260806 02:45:05.276973 28923 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } }
I20260806 02:45:05.282011 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.243027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.28.191.60:45379)
0806 02:45:05.243148 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.243158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.257726 (+ 14568us) server_negotiation.cc:407] Connection header received
0806 02:45:05.258303 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.258307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.258353 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.258413 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.260403 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.261230 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.265837 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.266186 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.266283 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.269719 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.269725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.269730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.269766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.271874 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.274843 (+ 2969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.274848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.274849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.274946 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.280371 (+ 5425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.280377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.280381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.280683 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.280854 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.281563 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.281738 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:45:05.283926 28888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91d20a8f1e5e456388822ddf43361515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6c22f121125462dadf753d327dbbdc6"
I20260806 02:45:05.284062 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.241904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.28.191.62:46397)
0806 02:45:05.242012 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.242017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.257478 (+ 15461us) server_negotiation.cc:407] Connection header received
0806 02:45:05.258029 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.258033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.258092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.258159 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.261391 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.262336 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.264474 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.264848 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.264935 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.272056 (+ 7121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.272060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.272063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.272095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.273866 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.275082 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.275085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.275087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.275174 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.275797 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.275804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.275808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.276247 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.276398 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.283660 (+ 7262us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.283842 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 02:45:05.285245 28986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:05.285329 28986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:05.285988 28888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:05.287375 28888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d20a8f1e5e456388822ddf43361515 in term 1.
I20260806 02:45:05.287449 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:05.241448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.28.191.61:34699)
0806 02:45:05.241589 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.241593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.246415 (+ 4822us) server_negotiation.cc:407] Connection header received
0806 02:45:05.246633 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.246639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.246699 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.246758 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.248034 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.248926 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.250532 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:05.250862 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:05.250941 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:05.277671 (+ 26730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:05.277678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:05.277682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:05.277723 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:05.279737 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.285566 (+ 5829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.285574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.285578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.285689 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:05.286187 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:05.286193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:05.286198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:05.286478 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:05.286644 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:05.287056 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:45:05.287209 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:05.288164 28923 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:05.294745 28888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [term 1 FOLLOWER]: Refusing update from remote peer 91d20a8f1e5e456388822ddf43361515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:05.295472 28987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:05.296927 28778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [term 1 FOLLOWER]: Refusing update from remote peer 91d20a8f1e5e456388822ddf43361515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:05.299106 28987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:05.303090 28961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d20a8f1e5e456388822ddf43361515. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.303202 28961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:45:05.306241 29013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:05.306331 29013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:05.309461 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.309634 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.309736 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d20a8f1e5e456388822ddf43361515. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.309819 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.314896 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.315040 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:05.316426 28961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.316550 28961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.317090 28996 mvcc.cc:205] Tried to move back new op lower bound from 7315391714479173632 to 7315391714288300032. Current Snapshot: MvccSnapshot[applied={T|T < 7315391714479173632}]
I20260806 02:45:05.317456 29019 mvcc.cc:205] Tried to move back new op lower bound from 7315391714479173632 to 7315391714288300032. Current Snapshot: MvccSnapshot[applied={T|T < 7315391714479173632}]
I20260806 02:45:05.319392 28936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.319541 28936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:05.323674 29014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91d20a8f1e5e456388822ddf43361515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d20a8f1e5e456388822ddf43361515" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 46397 } } peers { permanent_uuid: "0e58f01be8cd41d9a3a5844e8a4d0163" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 34699 } } peers { permanent_uuid: "f6c22f121125462dadf753d327dbbdc6" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 45379 } } }
I20260806 02:45:05.323840 29014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:05.324177 29016 mvcc.cc:205] Tried to move back new op lower bound from 7315391714479173632 to 7315391714288300032. Current Snapshot: MvccSnapshot[applied={T|T < 7315391714479173632}]
I20260806 02:45:05.324656 28989 catalog_manager.cc:1347] Generated new cluster ID: fd756d62294249efbe7da08f8cc8c8a2
I20260806 02:45:05.324726 28989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:05.350816 28989 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:05.352217 28989 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:05.362335 28989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91d20a8f1e5e456388822ddf43361515: Generated new TSK 0
I20260806 02:45:05.363174 28989 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:45:05.395146 28933 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.391290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.28.191.62:46397)
0806 02:45:05.391435 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.391440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.393648 (+ 2208us) server_negotiation.cc:407] Connection header received
0806 02:45:05.393874 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.393878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.393936 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.394003 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.394909 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54090: BlockingRecv error: recv got EOF from 127.0.0.1:54090 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 02:45:05.397186 28931 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.392132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.28.191.60:45379)
0806 02:45:05.392258 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.392263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.395887 (+ 3624us) server_negotiation.cc:407] Connection header received
0806 02:45:05.396305 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.396310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.396365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.396424 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.397021 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52656: BlockingRecv error: recv got EOF from 127.0.0.1:52656 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 02:45:05.398604 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.392937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.28.191.61:34699)
0806 02:45:05.393097 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.393102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.396214 (+ 3112us) server_negotiation.cc:407] Connection header received
0806 02:45:05.397745 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.397749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.397808 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.397865 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.398464 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46812: BlockingRecv error: recv got EOF from 127.0.0.1:46812 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 02:45:05.404162 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.402433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.28.191.61:34699)
0806 02:45:05.402565 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.402570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.403046 (+ 476us) server_negotiation.cc:407] Connection header received
0806 02:45:05.403301 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.403305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.403393 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.403456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.404050 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46820: BlockingRecv error: recv got EOF from 127.0.0.1:46820 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:45:05.409970 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.408230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.28.191.61:34699)
0806 02:45:05.408346 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.408350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.408906 (+ 556us) server_negotiation.cc:407] Connection header received
0806 02:45:05.409161 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.409165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.409227 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.409290 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.409857 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46824: BlockingRecv error: recv got EOF from 127.0.0.1:46824 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 02:45:05.417670 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.415886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.28.191.61:34699)
0806 02:45:05.416002 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.416007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.416544 (+ 537us) server_negotiation.cc:407] Connection header received
0806 02:45:05.416786 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.416790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.416849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.416906 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.417547 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46830: BlockingRecv error: recv got EOF from 127.0.0.1:46830 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:45:05.426215 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.424378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.28.191.61:34699)
0806 02:45:05.424562 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.424567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.425143 (+ 576us) server_negotiation.cc:407] Connection header received
0806 02:45:05.425366 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.425370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.425428 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.425500 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.426091 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46840: BlockingRecv error: recv got EOF from 127.0.0.1:46840 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260806 02:45:05.439073 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.437167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.28.191.61:34699)
0806 02:45:05.437298 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.437303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.437932 (+ 629us) server_negotiation.cc:407] Connection header received
0806 02:45:05.438178 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.438183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.438248 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.438309 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.438944 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46848: BlockingRecv error: recv got EOF from 127.0.0.1:46848 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:45:05.459597 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.458175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.28.191.61:34699)
0806 02:45:05.458320 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.458325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.458563 (+ 238us) server_negotiation.cc:407] Connection header received
0806 02:45:05.458715 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.458720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.458790 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.458905 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.459457 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46858: BlockingRecv error: recv got EOF from 127.0.0.1:46858 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:45:05.495750 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.494325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.28.191.61:34699)
0806 02:45:05.494468 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.494473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.494702 (+ 229us) server_negotiation.cc:407] Connection header received
0806 02:45:05.494839 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.494843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.494909 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.495027 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.495594 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46874: BlockingRecv error: recv got EOF from 127.0.0.1:46874 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:45:05.563887 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.562494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.28.191.61:34699)
0806 02:45:05.562712 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.562718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.562868 (+ 150us) server_negotiation.cc:407] Connection header received
0806 02:45:05.563061 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.563064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.563145 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.563244 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.563755 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46880: BlockingRecv error: recv got EOF from 127.0.0.1:46880 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260806 02:45:05.695755 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.694352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.28.191.61:34699)
0806 02:45:05.694515 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.694521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.694734 (+ 213us) server_negotiation.cc:407] Connection header received
0806 02:45:05.694922 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.694926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.694993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.695099 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.695624 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46894: BlockingRecv error: recv got EOF from 127.0.0.1:46894 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:45:05.955250 28820 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:05.954145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.28.191.61:34699)
0806 02:45:05.954283 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:05.954287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:05.954344 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:45:05.954488 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:05.954491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:05.954542 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:05.954645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:05.955132 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46906: BlockingRecv error: recv got EOF from 127.0.0.1:46906 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:45:06.286696 28986 catalog_manager.cc:1259] Loaded cluster ID: fd756d62294249efbe7da08f8cc8c8a2
I20260806 02:45:06.286806 28986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: loading cluster ID for follower catalog manager: success
I20260806 02:45:06.288565 28986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: acquiring CA information for follower catalog manager: success
I20260806 02:45:06.289935 28986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e58f01be8cd41d9a3a5844e8a4d0163: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:45:06.307792 29013 catalog_manager.cc:1259] Loaded cluster ID: fd756d62294249efbe7da08f8cc8c8a2
I20260806 02:45:06.307883 29013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: loading cluster ID for follower catalog manager: success
I20260806 02:45:06.309691 29013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: acquiring CA information for follower catalog manager: success
I20260806 02:45:06.310860 29013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6c22f121125462dadf753d327dbbdc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:45:06.473884 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.469896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.28.191.61:34699)
0806 02:45:06.470197 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.470200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.470216 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:06.470270 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.470273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.470403 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.470507 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:06.470776 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.471297 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.472228 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.472444 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.472519 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.472819 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:06.473365 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:06.473461 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.473566 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.473618 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":140,"threads_started":1}
I20260806 02:45:06.480954 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.477385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.28.191.60:45379)
0806 02:45:06.477660 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.477663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.477681 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:45:06.477761 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.477763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.477876 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.477992 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:06.478179 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.478737 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.479517 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.479762 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.479846 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.480107 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:06.480595 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:06.480692 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.480779 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.480830 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260806 02:45:06.488175 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.484493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.28.191.60:45379)
0806 02:45:06.484613 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.484618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.484820 (+ 202us) server_negotiation.cc:407] Connection header received
0806 02:45:06.484865 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.484868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.484914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.484986 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:06.485277 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.486017 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487092 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.487311 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487423 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.487674 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:06.487793 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:06.487883 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.487975 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.488025 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:45:06.488372 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.484132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.28.191.61:34699)
0806 02:45:06.484308 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.484313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.484669 (+ 356us) server_negotiation.cc:407] Connection header received
0806 02:45:06.484867 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.484870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.484917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.484964 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:06.485417 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.485970 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487104 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.487456 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487526 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.487880 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:06.487958 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:06.488005 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.488181 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.488248 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:45:06.489496 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.484066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.28.191.62:46397)
0806 02:45:06.484366 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.484370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.484613 (+ 243us) server_negotiation.cc:407] Connection header received
0806 02:45:06.484884 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.484887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.485040 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.485194 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:06.485586 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.486376 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487482 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.487842 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.487932 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.488360 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:06.488800 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:06.488907 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.489269 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.489334 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260806 02:45:06.492566 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28599
I20260806 02:45:06.508183 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28716
I20260806 02:45:06.521440 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 28830
2026-08-06T02:45:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3400 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:06 dist-test-slave-tfpf krb5kdc[29137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:06 dist-test-slave-tfpf krb5kdc[29137](info): set up 2 sockets
Aug 06 02:45:06 dist-test-slave-tfpf krb5kdc[29137](info): commencing operation
krb5kdc: starting...
W20260806 02:45:07.775707 29436 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/test-user.keytab.
Aug 06 02:45:07 dist-test-slave-tfpf krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.191.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
Entry for principal kudu/127.28.191.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab.
I20260806 02:45:07.912855 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:44409
--webserver_interface=127.28.191.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:08.029973 29156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:08.030350 29156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:08.030436 29156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:08.030522 29156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:08.034564 29156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:08.034675 29156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:08.034716 29156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:08.034754 29156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:08.034794 29156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:08.040416 29156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.62.keytab
--principal=kudu/127.28.191.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.62:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.191.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:08.041684 29156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:08.042843 29156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:08.051973 29162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.052043 29165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.051992 29167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:08.052448 29156 server_base.cc:1034] running on GCE node
I20260806 02:45:08.053131 29156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:08.054308 29156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:08.055507 29156 hybrid_clock.cc:648] HybridClock initialized: now 1785984308055497 us; error 31 us; skew 500 ppm
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:08.058782 29156 init.cc:377] Logged in from keytab as kudu/127.28.191.62@KRBTEST.COM (short username kudu)
I20260806 02:45:08.059866 29156 webserver.cc:460] Webserver started at http://127.28.191.62:45481/ using document root <none> and password file <none>
I20260806 02:45:08.060478 29156 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:08.060580 29156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:08.060834 29156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:08.063135 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data/instance:
uuid: "4aa43bba554c4ed690e7172b7a6201fc"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "0770ee4dbae571517f53d7a8b907c8f8"
server_key_iv: "4229b0795c7cf6b43556e0a558313039"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.063778 29156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance:
uuid: "4aa43bba554c4ed690e7172b7a6201fc"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "0770ee4dbae571517f53d7a8b907c8f8"
server_key_iv: "4229b0795c7cf6b43556e0a558313039"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.067868 29156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:08.070459 29176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.071638 29156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:08.071805 29156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal
uuid: "4aa43bba554c4ed690e7172b7a6201fc"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "0770ee4dbae571517f53d7a8b907c8f8"
server_key_iv: "4229b0795c7cf6b43556e0a558313039"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.071923 29156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:08.095561 29156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:08.099215 29156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:08.099444 29156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:08.106743 29156 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.62:44409
I20260806 02:45:08.107069 29253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.62:44409 every 8 connection(s)
I20260806 02:45:08.107777 29156 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:08.110653 29255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:08.111163 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29156
I20260806 02:45:08.111304 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-0/wal/instance
I20260806 02:45:08.111654 29436 external_mini_cluster.cc:1428] Setting key 2d5ac46790cf5b7b5579fd82932de2d2
I20260806 02:45:08.115507 29255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.191.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.61@KRBTEST.COM" created.
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:08.132292 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.117872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44409 (local address 127.0.0.1:41320)
0806 02:45:08.118446 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.118509 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.119684 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.120423 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.120431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.120946 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.121369 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.121394 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.122706 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.122709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.123466 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.123509 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.124190 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.124193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.124341 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.125015 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.125032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.126812 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.128948 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.128961 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.128975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.129393 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.129742 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.129746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.129748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.129871 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.130501 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.130508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.130665 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.130836 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.131250 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":112,"thread_start_us":111,"threads_started":1}
I20260806 02:45:08.132292 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.118588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.28.191.62:44409)
0806 02:45:08.118856 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.118866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.118900 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:45:08.119999 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.120003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.120189 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.120340 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.121547 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.122563 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.123629 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.124001 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.124203 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.126990 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.126999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.127014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.127047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.128804 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.129506 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.129511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.129512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.129622 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.129970 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.129973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.129977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.130378 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.130533 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.130924 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.131194 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260806 02:45:08.134967 29255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.135771 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.135503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35412)
0806 02:45:08.135622 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.135706 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:45:08.136039 29183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
I20260806 02:45:08.136929 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.136703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35428)
0806 02:45:08.136816 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.136874 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:45:08.137149 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:42045: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
Entry for principal kudu/127.28.191.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab.
I20260806 02:45:08.141894 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:42045
--webserver_interface=127.28.191.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:08.186398 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } attempt: 1
I20260806 02:45:08.187485 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.187099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35442)
0806 02:45:08.187291 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.187372 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:45:08.188382 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.188132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35456)
0806 02:45:08.188254 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.188312 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:45:08.188548 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:42045: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
I20260806 02:45:08.256758 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } attempt: 2
I20260806 02:45:08.257709 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.257360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35466)
0806 02:45:08.257527 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.257615 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:45:08.258584 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.258280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35480)
0806 02:45:08.258420 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.258488 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:45:08.258761 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.61:42045: Network error: Client connection negotiation failed: client connection to 127.28.191.61:42045: connect: Connection refused (error 111)
W20260806 02:45:08.261430 29262 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:08.261730 29262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:08.261785 29262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:08.261826 29262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:08.265926 29262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:08.265981 29262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:08.266001 29262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:08.266074 29262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:08.266115 29262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:08.271607 29262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.61.keytab
--principal=kudu/127.28.191.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.61:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.191.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:08.272801 29262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:08.273972 29262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:08.282765 29271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.282770 29273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.282768 29269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:08.282873 29262 server_base.cc:1034] running on GCE node
I20260806 02:45:08.283584 29262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:08.284602 29262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:08.285821 29262 hybrid_clock.cc:648] HybridClock initialized: now 1785984308285805 us; error 26 us; skew 500 ppm
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:08.288686 29262 init.cc:377] Logged in from keytab as kudu/127.28.191.61@KRBTEST.COM (short username kudu)
I20260806 02:45:08.289731 29262 webserver.cc:460] Webserver started at http://127.28.191.61:33491/ using document root <none> and password file <none>
I20260806 02:45:08.290331 29262 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:08.290427 29262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:08.290670 29262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:08.293038 29262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data/instance:
uuid: "89a73a51d8244a2aa63460ed51407472"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "baefa862ef9dacf3abe4eee0a857d9c5"
server_key_iv: "3e80b0389a2f44de50fd58c39dcb5b3d"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.293933 29262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal/instance:
uuid: "89a73a51d8244a2aa63460ed51407472"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "baefa862ef9dacf3abe4eee0a857d9c5"
server_key_iv: "3e80b0389a2f44de50fd58c39dcb5b3d"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.298031 29262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:08.300745 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.302258 29262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:45:08.302430 29262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal
uuid: "89a73a51d8244a2aa63460ed51407472"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "baefa862ef9dacf3abe4eee0a857d9c5"
server_key_iv: "3e80b0389a2f44de50fd58c39dcb5b3d"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.302603 29262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:08.333616 29262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:08.337013 29262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:08.337278 29262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:08.345402 29262 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.61:42045
I20260806 02:45:08.345705 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.61:42045 every 8 connection(s)
I20260806 02:45:08.346532 29262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/data/info.pb
I20260806 02:45:08.349485 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:08.349905 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } attempt: 3
I20260806 02:45:08.350932 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29262
I20260806 02:45:08.351063 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-1/wal/instance
I20260806 02:45:08.351373 29436 external_mini_cluster.cc:1428] Setting key 90c58248c5b786d981cec4ca827df3ef
I20260806 02:45:08.355625 29355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.191.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.191.60@KRBTEST.COM" created.
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:08.365785 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.350500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35482)
0806 02:45:08.350668 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.350686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.350783 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.353525 (+ 2742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.353529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.353765 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.353926 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.353935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.356272 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.356278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.357310 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.357328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.358535 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.358539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.358674 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.359275 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.359294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.360765 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.363451 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.363454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.363457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.363751 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.364186 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.364189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.364191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.364287 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.365070 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.365095 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.365228 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.365331 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.365558 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:45:08.367609 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.351279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.28.191.61:42045)
0806 02:45:08.351834 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.351853 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.351896 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:45:08.352654 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.352687 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.353169 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.354161 (+ 992us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.354619 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.356095 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.357497 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.358386 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.358600 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.360989 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.360996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.361010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.361048 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.363315 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.363884 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.363890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.363897 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.364086 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.364419 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.364422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.364424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.364931 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.365591 (+ 660us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.366056 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.366397 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":202,"threads_started":1}
I20260806 02:45:08.369369 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.357531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.28.191.62:44409)
0806 02:45:08.357669 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.357675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.358024 (+ 349us) server_negotiation.cc:407] Connection header received
0806 02:45:08.358231 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.358237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.358297 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.358340 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.359350 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.360211 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.361456 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.362490 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.362633 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.365373 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.365377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.365380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.365412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.367496 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.368060 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.368063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.368064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.368139 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.368427 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.368429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.368430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.368588 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.368701 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.369110 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.369233 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:45:08.369494 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.357516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44409 (local address 127.0.0.1:41336)
0806 02:45:08.357903 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.357953 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.358109 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.358531 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.358536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.358997 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.359184 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.359191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.360357 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.360361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.361306 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.361318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.362624 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.362628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.362749 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.363232 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.363244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.365227 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.367620 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.367632 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.367640 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.367953 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.368236 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.368238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.368240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.368335 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.368692 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.368698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.368837 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.368997 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.369235 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":150,"threads_started":1}
I20260806 02:45:08.370011 29255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.370683 29355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.370748 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.370534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53202)
0806 02:45:08.370636 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.370686 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 02:45:08.371609 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.371323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53206)
0806 02:45:08.371492 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.371540 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 02:45:08.371814 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:08.379985 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.371151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35492)
0806 02:45:08.371325 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.371341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.371416 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.371688 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.371694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.371886 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.372033 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.372045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.372882 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.372886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.373888 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.373902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.374360 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.374366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.374494 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.375165 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.375183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.376628 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.378337 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.378340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.378341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.378574 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.378882 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.378889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.378891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.379008 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.379481 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.379483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.379563 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.379648 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.379812 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 02:45:08.379994 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.371256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.28.191.61:42045)
0806 02:45:08.371361 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.371366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.371380 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:08.371518 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.371523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.371568 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.371656 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.372171 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.372758 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.374013 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.374235 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.374315 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.376788 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.376795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.376800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.376828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.378239 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.378704 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.378707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.378708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.378780 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.379110 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.379114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.379115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.379380 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.379488 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.379711 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.379877 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":43}
I20260806 02:45:08.380738 29355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.381556 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.381317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53212)
0806 02:45:08.381429 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.381505 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:45:08.382267 29290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
Entry for principal kudu/127.28.191.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab.
I20260806 02:45:08.383235 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.382973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53226)
0806 02:45:08.383092 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.383157 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:45:08.383387 29355 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
I20260806 02:45:08.384081 29436 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
/tmp/dist-test-tasklwD5TR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:37569
--webserver_interface=127.28.191.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.191.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:08.402987 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 1
I20260806 02:45:08.403931 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.403624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53230)
0806 02:45:08.403785 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.403847 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:45:08.404671 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.404454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53242)
0806 02:45:08.404565 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.404618 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:45:08.404803 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
I20260806 02:45:08.432698 29355 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 1
I20260806 02:45:08.433698 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.433345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53252)
0806 02:45:08.433519 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.433603 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:45:08.434461 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.434231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53258)
0806 02:45:08.434342 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.434395 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:45:08.434608 29355 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
I20260806 02:45:08.479984 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 2
I20260806 02:45:08.480855 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.480537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53270)
0806 02:45:08.480699 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.480769 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:45:08.481685 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.481451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53272)
0806 02:45:08.481564 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.481617 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:45:08.481869 29255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
I20260806 02:45:08.502815 29355 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 2
I20260806 02:45:08.503799 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.503387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53276)
0806 02:45:08.503624 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.503704 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 02:45:08.503516 29367 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:45:08.503926 29367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:08.503990 29367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:08.504072 29367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:45:08.504637 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.504379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53292)
0806 02:45:08.504497 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.504570 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:45:08.504829 29355 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.191.60:37569: Network error: Client connection negotiation failed: client connection to 127.28.191.60:37569: connect: Connection refused (error 111)
W20260806 02:45:08.509164 29367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:08.509243 29367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:08.509294 29367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:08.509335 29367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:08.509378 29367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:08.515174 29367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.191.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.191.62:44409,127.28.191.61:42045,127.28.191.60:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/krb5kdc/kudu_127.28.191.60.keytab
--principal=kudu/127.28.191.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.191.60:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.191.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:08.516319 29367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:08.517455 29367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:08.526016 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.526016 29373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.526041 29372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:08.526340 29367 server_base.cc:1034] running on GCE node
I20260806 02:45:08.527038 29367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:08.528178 29367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:08.529381 29367 hybrid_clock.cc:648] HybridClock initialized: now 1785984308529361 us; error 37 us; skew 500 ppm
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:08.532343 29367 init.cc:377] Logged in from keytab as kudu/127.28.191.60@KRBTEST.COM (short username kudu)
I20260806 02:45:08.533295 29367 webserver.cc:460] Webserver started at http://127.28.191.60:39525/ using document root <none> and password file <none>
I20260806 02:45:08.533845 29367 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:08.533910 29367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:08.534088 29367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:08.536139 29367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data/instance:
uuid: "7aed091b313a4ac69d06d12d6fd67496"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "c33e881f2597e8ca5fcfdf288d40b120"
server_key_iv: "b7cb0f989b9ba56a8cf8e3b68a7e4b98"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.536685 29367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal/instance:
uuid: "7aed091b313a4ac69d06d12d6fd67496"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "c33e881f2597e8ca5fcfdf288d40b120"
server_key_iv: "b7cb0f989b9ba56a8cf8e3b68a7e4b98"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.540448 29367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 02:45:08.542971 29386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.544205 29367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:45:08.544309 29367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data,/tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal
uuid: "7aed091b313a4ac69d06d12d6fd67496"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-tfpf"
server_key: "c33e881f2597e8ca5fcfdf288d40b120"
server_key_iv: "b7cb0f989b9ba56a8cf8e3b68a7e4b98"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.544399 29367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:08.560070 29367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:08.563514 29367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:08.563700 29367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:08.570910 29367 rpc_server.cc:307] RPC server started. Bound to: 127.28.191.60:37569
I20260806 02:45:08.571324 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.191.60:37569 every 8 connection(s)
I20260806 02:45:08.571995 29367 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/data/info.pb
I20260806 02:45:08.574918 29471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:08.578979 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.581040 29255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 3
I20260806 02:45:08.581712 29436 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu as pid 29367
I20260806 02:45:08.581861 29436 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklwD5TR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984016215184-29436-0/minicluster-data/master-2/wal/instance
I20260806 02:45:08.582197 29436 external_mini_cluster.cc:1428] Setting key e914a2350fbdc2e075e5f502a76a9b0a
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.62@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:45:08.594609 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.581508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53302)
0806 02:45:08.581706 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.581721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.581809 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.583607 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.583610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.583802 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.583877 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.583882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.585778 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.585783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.586638 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.586648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.587482 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.587485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.587569 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.588129 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.588144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.589805 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.592383 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.592385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.592387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.592688 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.593156 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.593172 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.593175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.593283 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.594091 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.594097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.594189 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.594292 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.594426 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 02:45:08.595059 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.580889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.28.191.62:44409)
0806 02:45:08.581100 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.581107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.582425 (+ 1318us) server_negotiation.cc:407] Connection header received
0806 02:45:08.583379 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.583382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.583424 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.583526 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.584626 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.585391 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.586931 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.587224 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.587339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.590712 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.590715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.590718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.590750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.592763 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.593388 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.593391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.593393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.593477 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.593787 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.593790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.593792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.594020 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.594140 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.594786 (+ 646us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.594919 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:08.595891 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.581350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.62:44409 (local address 127.0.0.1:41340)
0806 02:45:08.582271 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.582327 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.583158 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.583556 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.583567 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.583993 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.584449 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.584467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.585507 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.585513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.586772 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.586788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.587388 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.587393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.587586 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.588311 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.588326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.590528 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.592871 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.592879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.592888 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.593289 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.593576 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.593582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.593586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.593691 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.594197 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.594203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.594355 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.594497 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.594809 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":201,"spinlock_wait_cycles":4736,"thread_start_us":150,"threads_started":1}
I20260806 02:45:08.595988 29355 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } attempt: 3
I20260806 02:45:08.595911 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.582853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.28.191.60:37569)
0806 02:45:08.583090 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.583100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.583290 (+ 190us) server_negotiation.cc:407] Connection header received
0806 02:45:08.583450 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.583453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.583499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.583623 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.584926 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.586367 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.587547 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.587900 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.588022 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.591199 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.591205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.591212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.591234 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.593750 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.594232 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.594236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.594237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.594320 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.594622 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.594625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.594626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.594836 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.594983 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.595638 (+ 655us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.595747 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":738,"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260806 02:45:08.597337 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.581681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.28.191.60:37569)
0806 02:45:08.582148 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.582166 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.582204 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:45:08.583158 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.583178 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.583435 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.583967 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.584197 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.585666 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.586778 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.587335 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.587579 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.589944 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.589950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.589963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.590035 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.592105 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.592806 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.592819 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.592843 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.593005 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.593408 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.593412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.593414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.593976 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.594116 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.594431 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.597222 (+ 2791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":351,"spinlock_wait_cycles":2176,"thread_start_us":169,"threads_started":1}
I20260806 02:45:08.597661 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.601317 29255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: Bootstrap starting.
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.61@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:08.604404 29255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: Neither blocks nor log segments found. Creating new log.
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.62@KRBTEST.COM
I20260806 02:45:08.605427 29255 log.cc:824] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:08.607538 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.598103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.61:42045 (local address 127.0.0.1:35508)
0806 02:45:08.598560 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.598576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.598701 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.599042 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.599045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.599225 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.599337 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.599344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.600305 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.600308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.601033 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.601177 (+ 144us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.601827 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.601830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.601921 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.602359 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.602374 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.603770 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.605847 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.605850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.605852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.606098 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.606393 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.606396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.606398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.606501 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.607018 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.607021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.607118 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.607237 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.607364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389}
I20260806 02:45:08.608063 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.598188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.28.191.61:42045)
0806 02:45:08.598300 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.598305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.598628 (+ 323us) server_negotiation.cc:407] Connection header received
0806 02:45:08.598810 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.598813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.598855 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.598903 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.599467 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.600155 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.601303 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.601590 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.601679 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.603922 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.603926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.603928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.603952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.605714 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.606210 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.606213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.606215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.606304 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.606610 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.606613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.606615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.606880 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.607019 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.607801 (+ 782us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.607923 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:45:08.608268 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.596456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53334)
0806 02:45:08.596563 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.596580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.596666 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.597100 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.597105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.597287 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.597393 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.597402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.598948 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.598952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.599756 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.599766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.600428 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.600431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.600512 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.601013 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.601027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.602917 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.605890 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.605894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.605898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.606132 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.606801 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.606804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.606806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.606897 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.607681 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.607686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.607782 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.607865 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.608063 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 02:45:08.608438 29255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: No bootstrap required, opened a new log
I20260806 02:45:08.608670 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:08.608907 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.596731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.28.191.60:37569)
0806 02:45:08.596834 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.596838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.596850 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:08.596882 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.596884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.596922 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.597049 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.598150 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.598817 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.599899 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.600287 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.603048 (+ 2761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.603156 (+ 108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.603161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.603166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.603470 (+ 304us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.605760 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.606597 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.606600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.606602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.606688 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.607277 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.607283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.607285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.607499 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.608543 (+ 1044us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.608625 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.608734 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:45:08.610828 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.599123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.28.191.62:44409)
0806 02:45:08.599218 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.599223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.599377 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:45:08.599542 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.599545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.599585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.599630 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.600434 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.601191 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.602360 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.602718 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.602793 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.605985 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.605991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.605997 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.606025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.608684 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.609399 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.609402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.609404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.609507 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.609828 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.609832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.609833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.610041 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.610195 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.610557 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.610675 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:45:08.612048 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: Bootstrap starting.
I20260806 02:45:08.613374 29255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.613600 29255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa43bba554c4ed690e7172b7a6201fc, State: Initialized, Role: FOLLOWER
I20260806 02:45:08.613961 29255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.191.60@KRBTEST.COM for kudu/127.28.191.60@KRBTEST.COM
I20260806 02:45:08.614468 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.614619 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.614842 29355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:08.615442 29255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:08.615808 29355 log.cc:824] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:08.617908 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.609283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.191.60:37569 (local address 127.0.0.1:53340)
0806 02:45:08.609566 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.609581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.609650 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.609940 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.609945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.610119 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.610201 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.610207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.611155 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.611159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.611863 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.611873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.612391 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.612395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.612464 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.612912 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.612925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.614441 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.616372 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.616375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.616377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.616615 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.616919 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.616925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.616927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.617025 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.617488 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.617491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.617567 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.617638 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.617750 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260806 02:45:08.618498 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.609476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.28.191.60:37569)
0806 02:45:08.609764 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.609768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.609777 (+ 9us) server_negotiation.cc:407] Connection header received
0806 02:45:08.609804 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.609806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.609840 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.610296 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.610357 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.611045 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.611995 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.612278 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.613553 (+ 1275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.614573 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.614576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.614578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.614599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.616259 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.616712 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.616715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.616717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.616812 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.617196 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.617201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.617205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.617388 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.618195 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.618272 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.618375 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260806 02:45:08.619427 29355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: No bootstrap required, opened a new log
Aug 06 02:45:08 dist-test-slave-tfpf krb5kdc[29137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.191.61@KRBTEST.COM
I20260806 02:45:08.620723 29156 master_runner.cc:186] Error getting master registration for 127.28.191.61:42045: OK, instance_id { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" instance_seqno: 1785984308343054 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:45:08.624035 29156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:45:08.624290 29355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.624547 29355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a73a51d8244a2aa63460ed51407472, State: Initialized, Role: FOLLOWER
I20260806 02:45:08.624904 29355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.625502 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.625707 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.626394 29355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:08.626427 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.616290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.28.191.61:42045)
0806 02:45:08.616401 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.616407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.616422 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:08.616459 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.616462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.616516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.616595 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.617480 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.618197 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.619319 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.619615 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.619722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.622593 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.622597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.622599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.622626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.624227 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.624729 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.624733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.624735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.624819 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.625167 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.625171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.625173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.625395 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.625556 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.626176 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.626286 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:45:08.626478 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: Bootstrap starting.
I20260806 02:45:08.629807 29436 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:08.630107 29471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:08.631075 29471 log.cc:824] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:08.631194 29220 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4aa43bba554c4ed690e7172b7a6201fc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41342
I20260806 02:45:08.631409 29220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:45:08.631503 29220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:45:08.632191 29506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:08.632261 29506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:08.632958 29220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.633859 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: No bootstrap required, opened a new log
I20260806 02:45:08.633993 29220 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [CANDIDATE]: Term 1 election: Requested vote from peers 89a73a51d8244a2aa63460ed51407472 (127.28.191.61:42045), 7aed091b313a4ac69d06d12d6fd67496 (127.28.191.60:37569)
I20260806 02:45:08.634948 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4aa43bba554c4ed690e7172b7a6201fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "89a73a51d8244a2aa63460ed51407472"
I20260806 02:45:08.635192 29322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:45:08.635406 29492 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:08.635529 29492 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:08.636536 29322 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa43bba554c4ed690e7172b7a6201fc in term 1.
I20260806 02:45:08.636986 29183 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa43bba554c4ed690e7172b7a6201fc, 89a73a51d8244a2aa63460ed51407472; no voters:
I20260806 02:45:08.638851 29479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:08.639086 29479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [term 1 LEADER]: Becoming Leader. State: Replica: 4aa43bba554c4ed690e7172b7a6201fc, State: Running, Role: LEADER
I20260806 02:45:08.639446 29479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.641309 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa43bba554c4ed690e7172b7a6201fc. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.641419 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:08.643435 29524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:08.640822 29262 master_runner.cc:186] Error getting master registration for 127.28.191.60:37569: OK, instance_id { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" instance_seqno: 1785984308568882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:45:08.644381 29524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:08.644454 29262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:45:08.645532 29471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.645792 29471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aed091b313a4ac69d06d12d6fd67496, State: Initialized, Role: FOLLOWER
I20260806 02:45:08.646121 29471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } }
I20260806 02:45:08.647568 29471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:08.647642 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4aa43bba554c4ed690e7172b7a6201fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7aed091b313a4ac69d06d12d6fd67496"
I20260806 02:45:08.647874 29430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:08.650622 29430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa43bba554c4ed690e7172b7a6201fc in term 1.
I20260806 02:45:08.654044 29529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.656759 29529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.656937 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.644590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.28.191.60:37569)
0806 02:45:08.644716 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.644720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.646693 (+ 1973us) server_negotiation.cc:407] Connection header received
0806 02:45:08.646866 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.646869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.646914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.646964 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.648155 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.648874 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.650359 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.650752 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.650843 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.653359 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.653364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.653368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.653396 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.655120 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.655589 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.655592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.655594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.655678 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.655979 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.655982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.655984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.656193 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.656300 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.656640 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.656767 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
I20260806 02:45:08.658161 29430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [term 1 FOLLOWER]: Refusing update from remote peer 4aa43bba554c4ed690e7172b7a6201fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:08.658340 29322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [term 1 FOLLOWER]: Refusing update from remote peer 4aa43bba554c4ed690e7172b7a6201fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:08.658829 29521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:08.659541 29479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:08.662022 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.639373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.28.191.62:44409)
0806 02:45:08.641656 (+ 2283us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.641662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.643465 (+ 1803us) server_negotiation.cc:407] Connection header received
0806 02:45:08.643840 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.643843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.643896 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.643952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.645798 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.646601 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.648931 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.649283 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.649391 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.651816 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.651820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.651823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.651855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.653617 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.660132 (+ 6515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.660135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.660137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.660304 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.660644 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.660648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.660650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.660884 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.661014 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.661748 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.661873 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205}
I20260806 02:45:08.666409 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa43bba554c4ed690e7172b7a6201fc. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.666548 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.666699 29529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa43bba554c4ed690e7172b7a6201fc. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.666821 29529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.667109 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.642790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.28.191.61:42045)
0806 02:45:08.642972 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.642978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.653350 (+ 10372us) server_negotiation.cc:407] Connection header received
0806 02:45:08.653570 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.653575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.653641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.653691 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.654613 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.655365 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.656486 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.656847 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.656929 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.661226 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.661230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.661232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.661258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.662925 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.665559 (+ 2634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.665564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.665565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.665662 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.665971 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.665975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.665977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.666239 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.666373 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.666768 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.666907 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:08.669486 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.669616 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:08.670668 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.670784 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.671136 29524 catalog_manager.cc:1347] Generated new cluster ID: 590e11c987234d9ea634b14dc7a50914
I20260806 02:45:08.671216 29524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:08.677465 29549 mvcc.cc:205] Tried to move back new op lower bound from 7315391728254496768 to 7315391728190832640. Current Snapshot: MvccSnapshot[applied={T|T < 7315391728254496768}]
W20260806 02:45:08.680403 29545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:08.680871 29545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:08.685098 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.685251 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:08.686483 29529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4aa43bba554c4ed690e7172b7a6201fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa43bba554c4ed690e7172b7a6201fc" member_type: VOTER last_known_addr { host: "127.28.191.62" port: 44409 } } peers { permanent_uuid: "89a73a51d8244a2aa63460ed51407472" member_type: VOTER last_known_addr { host: "127.28.191.61" port: 42045 } } peers { permanent_uuid: "7aed091b313a4ac69d06d12d6fd67496" member_type: VOTER last_known_addr { host: "127.28.191.60" port: 37569 } } }
I20260806 02:45:08.686604 29529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:08.691807 29524 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:08.697485 29524 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:08.710968 29524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa43bba554c4ed690e7172b7a6201fc: Generated new TSK 0
I20260806 02:45:08.711836 29524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:45:08.753870 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.750600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.28.191.61:42045)
0806 02:45:08.750734 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.750738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.752656 (+ 1918us) server_negotiation.cc:407] Connection header received
0806 02:45:08.752858 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.752862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.752919 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.752974 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.753728 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35532: BlockingRecv error: recv got EOF from 127.0.0.1:35532 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 02:45:08.755235 29475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.751445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.28.191.60:37569)
0806 02:45:08.751585 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.751589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.754358 (+ 2769us) server_negotiation.cc:407] Connection header received
0806 02:45:08.754535 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.754539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.754586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.754636 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.755116 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53354: BlockingRecv error: recv got EOF from 127.0.0.1:53354 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:45:08.756485 29259 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.752233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.28.191.62:44409)
0806 02:45:08.752366 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.752370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.755694 (+ 3324us) server_negotiation.cc:407] Connection header received
0806 02:45:08.755807 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.755811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.755869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.755956 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.756367 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41360: BlockingRecv error: recv got EOF from 127.0.0.1:41360 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:45:08.760846 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.759026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.28.191.61:42045)
0806 02:45:08.759134 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.759138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.759673 (+ 535us) server_negotiation.cc:407] Connection header received
0806 02:45:08.759902 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.759906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.759986 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.760069 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.760745 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35536: BlockingRecv error: recv got EOF from 127.0.0.1:35536 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 02:45:08.765923 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.764343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.28.191.61:42045)
0806 02:45:08.764438 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.764442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.764913 (+ 471us) server_negotiation.cc:407] Connection header received
0806 02:45:08.765200 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.765204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.765253 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.765305 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.765812 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35544: BlockingRecv error: recv got EOF from 127.0.0.1:35544 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260806 02:45:08.771654 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.770087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.28.191.61:42045)
0806 02:45:08.770182 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.770187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.770746 (+ 559us) server_negotiation.cc:407] Connection header received
0806 02:45:08.770928 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.770931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.770980 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.771031 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.771550 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35546: BlockingRecv error: recv got EOF from 127.0.0.1:35546 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 02:45:08.780642 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.777862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.28.191.61:42045)
0806 02:45:08.778007 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.778011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.779624 (+ 1613us) server_negotiation.cc:407] Connection header received
0806 02:45:08.779828 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.779832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.779904 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.779959 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.780533 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35552: BlockingRecv error: recv got EOF from 127.0.0.1:35552 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 02:45:08.792325 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.790732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.28.191.61:42045)
0806 02:45:08.790835 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.790838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.791379 (+ 541us) server_negotiation.cc:407] Connection header received
0806 02:45:08.791583 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.791586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.791639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.791691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.792214 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:45:08.812964 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.811004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.28.191.61:42045)
0806 02:45:08.811167 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.811171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.811865 (+ 694us) server_negotiation.cc:407] Connection header received
0806 02:45:08.812100 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.812103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.812162 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.812218 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.812840 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35572: BlockingRecv error: recv got EOF from 127.0.0.1:35572 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 02:45:08.849771 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.847713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.28.191.61:42045)
0806 02:45:08.847840 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.847845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.848560 (+ 715us) server_negotiation.cc:407] Connection header received
0806 02:45:08.848840 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.848847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.848918 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.848977 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.849632 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35574: BlockingRecv error: recv got EOF from 127.0.0.1:35574 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:45:08.918108 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:08.916637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.28.191.61:42045)
0806 02:45:08.916812 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.916817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.917056 (+ 239us) server_negotiation.cc:407] Connection header received
0806 02:45:08.917285 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.917296 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.917351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.917449 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.917957 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35578: BlockingRecv error: recv got EOF from 127.0.0.1:35578 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 02:45:09.049885 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:09.048532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.28.191.61:42045)
0806 02:45:09.048703 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.048708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.048905 (+ 197us) server_negotiation.cc:407] Connection header received
0806 02:45:09.049032 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.049035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.049131 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.049213 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:09.049737 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35586: BlockingRecv error: recv got EOF from 127.0.0.1:35586 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:45:09.309420 29360 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:09.308245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.28.191.61:42045)
0806 02:45:09.308422 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.308426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.308505 (+ 79us) server_negotiation.cc:407] Connection header received
0806 02:45:09.308662 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.308665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.308715 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.308788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:09.309286 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35596: BlockingRecv error: recv got EOF from 127.0.0.1:35596 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:45:09.633723 29506 catalog_manager.cc:1259] Loaded cluster ID: 590e11c987234d9ea634b14dc7a50914
I20260806 02:45:09.633816 29506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: loading cluster ID for follower catalog manager: success
I20260806 02:45:09.635536 29506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: acquiring CA information for follower catalog manager: success
I20260806 02:45:09.636760 29506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89a73a51d8244a2aa63460ed51407472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:45:09.682452 29545 catalog_manager.cc:1259] Loaded cluster ID: 590e11c987234d9ea634b14dc7a50914
I20260806 02:45:09.682541 29545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: loading cluster ID for follower catalog manager: success
I20260806 02:45:09.684419 29545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: acquiring CA information for follower catalog manager: success
I20260806 02:45:09.685722 29545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7aed091b313a4ac69d06d12d6fd67496: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:45:09.827812 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:09.823517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.28.191.61:42045)
0806 02:45:09.823805 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.823812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.823832 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:09.824021 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.824024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.824175 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.824300 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:09.824573 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.825141 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.826026 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.826266 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.826379 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:09.826810 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:09.827276 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:09.827440 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:09.827479 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 02:45:09.827569 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260806 02:45:09.835319 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:09.831146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.28.191.60:37569)
0806 02:45:09.831410 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.831414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.831430 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:09.831587 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.831590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.831733 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.831845 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:09.832102 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.832679 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.833677 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.833946 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.834043 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:09.834394 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:09.834855 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:09.834964 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:09.835018 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:45:09.835072 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260806 02:45:09.843492 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:09.838715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.28.191.61:42045)
0806 02:45:09.839003 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.839008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.839153 (+ 145us) server_negotiation.cc:407] Connection header received
0806 02:45:09.839349 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.839356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.839430 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.839484 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:09.839990 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.840820 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.842088 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.842449 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.842582 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:09.842925 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:09.843016 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:09.843107 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:09.843247 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:45:09.843313 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260806 02:45:09.843812 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:09.838852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.28.191.62:44409)
0806 02:45:09.839101 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.839105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.839231 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:45:09.839444 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.839448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.839623 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.839772 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:09.840039 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.840834 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.841905 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.842266 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.842394 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:09.842766 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:09.843162 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:09.843261 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:09.843551 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:45:09.843629 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260806 02:45:09.844136 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:09.838723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.28.191.60:37569)
0806 02:45:09.838866 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:09.838870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:09.839284 (+ 414us) server_negotiation.cc:407] Connection header received
0806 02:45:09.839462 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:09.839465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:09.839512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:09.839584 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:09.840979 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.841777 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.842890 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:09.843200 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:09.843282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:09.843670 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:09.843749 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:09.843796 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:09.843981 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:45:09.844043 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 02:45:09.848088 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29156
I20260806 02:45:09.860457 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29262
I20260806 02:45:09.872311 29436 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklwD5TR/build/debug/bin/kudu with pid 29367
2026-08-06T02:45:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3347 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6748 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293632 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST